Streamlining Releases: Building Scalable CICD Pipelines with Changesets

Recap

In our post on versioning, we laid the groundwork for understanding software package versioning — a key part of managing dependencies and tracking changes effectively.

We also explored the pros and cons of multirepo vs. monorepo architectures, and why we recommend starting with a monorepo setup for most teams.

In another deep dive, we unpacked strategies for handling independent vs. locked versioning in a shared workspace.

Now, in this post, we're bringing all those threads together. We'll walk through how to design a robust, scalable release and deployment pipeline that builds on the principles we've already discussed.

The Problem

Growing Pains in Scaling DevOps at Prosopo

At Prosopo, we've grown rapidly by delivering high-quality software at scale. But, as with any scaling software team, we've encountered our fair share of growing pains.

Our Initial Git Branching Strategy

From day one, we adopted a simple three-branch model:

main – The production branch. Any code pushed here is automatically released and deployed via CI/CD pipelines.
staging – Mirrors main, but deploys to a staging environment instead.
dev – Our bleeding-edge development branch, used for local development only. It’s not connected to any environment or CI/CD pipeline.

We initially opted for a multi-repo setup, but quickly switched to a monorepo. This transition helped alleviate early pains and significantly accelerated development.

What Worked (and What Didn’t)

This structure served us well in the early days. The staging branch gave us a clean environment for QA and testing, while main ensured seamless, automated production deployments. For a small, fast-moving team, it was ideal.

However, as our product matured and our engineering team expanded, cracks began to show:

Developers started bypassing the dev branch entirely, pushing and merging directly into staging.
staging became a bottleneck: every new feature and bug fix branched from and merged into it.
As a result, staging was rarely in a production-ready state. Partial features and incomplete fixes made it hard to ship anything without breaking something else.

This single-track development approach led to frequent blocking, frustration, and ultimately — hotfixes being made directly on main. While this sped up urgent fixes, it sometimes skipped QA entirely, leading to minor regressions.

The Cost of Speed

Because main was wired to auto-release, changes were deployed immediately — great for speed, terrible for rollbacks. We've had a few rough (thankfully minor) releases where a broken service couldn’t be easily reverted because our DevOps system doesn’t support quick rollbacks (i.e., no easy way to redeploy a previous version).

Versioning Woes

To complicate matters further, we use a locked versioning strategy. This means every package bump requires a coordinated update across the entire monorepo. In practice:

Staging's unstable state often led to version conflicts.
Once we could do a bump, everything had to be rebuilt, released, and deployed.
This process could take up to an hour, resulting in lost developer momentum due to context switching.

Where We Are Now

In short, things got messy. But this isn’t uncommon.

What started as a lightweight model — perfect for a small team and a new product — eventually buckled under scale. We successfully scaled our product and codebase, but not our DevOps infrastructure. We estimate this inefficiency has slowed us down by at least 5x, thanks to feature blocking and insufficient hotfix testing.

Our development velocity was dropping. We knew we had to act fast.

Down the rabbit hole we go...

Breaking It Down

Before we dive into solutions, it's important to deconstruct the challenge into smaller, actionable requirements. Here's what we need our system to support:

Automated testing – Every change should be validated by tests without manual intervention.
Automated deployments – Code should be pushed to environments seamlessly via CI/CD.
Automated versioning – Version bumps should be handled programmatically and consistently.
Automated publishing – New versions should be released to package registries (e.g., npmjs).
Manually triggered deployments – Even with automation, deployments should require manual confirmation to avoid accidental releases.
Parallel development – Teams should be able to work on multiple features or bug fixes simultaneously without blocking each other.
Version-based deployments – We need the ability to deploy any version of our software to any environment, on demand.
Dynamic environment creation – Spinning up new environments (e.g., for feature testing or dev previews) should be easy and fast.
Infrastructure flexibility – We should be able to swap out service hosts or providers (e.g., AWS → Google Cloud, one CDN → another) with minimal friction.
Rollback support – We need a way to revert to previous versions quickly and easily, without losing data or state.
Changelogs – Each package should have a changelog that documents all changes, features, and fixes in a clear and consistent format.

Solution

To address the challenges we outlined, here's the approach we've designed:

Branching Strategy

main – Continues to serve as our production branch.
staging – Remains our integration/testing branch for pre-production validation.
build/* – Any branch following this pattern is treated as a build branch, used for creating temporary or pre-release package versions.

This setup gives us clear boundaries between stable releases, staging changes, and experimental builds.

Versioning Strategy

We've adopted Changesets as our tool of choice for managing versioning across the monorepo. It offers a lightweight and flexible way to track and apply version bumps based on actual changes.

Other tools we considered include:

nx – Feature-rich but too opinionated and heavy for our use case.
lerna – Once popular, but now largely a wrapper around nx after becoming unmaintained.
rush – A strong alternative to changesets that we may explore further (we discovered it a bit too late in our implementation 🙃).

Changesets works by generating small markdown files that describe changes in one or more packages. For example:

---
"my-package": patch
---

Fixed a bug where the API would return incorrect data under certain conditions.

changesets simplifies version management by storing change descriptions in markdown files under the .changesets directory. For example:

/.changesets/
	change1.md
	change2.md
	change3.md

To create these files, changesets provides an interactive CLI tool. By running npx @changesets/cli add, users are guided through selecting the affected packages, specifying the type of change (patch, minor, or major), and providing a description. This ensures all changes are properly documented and versioned.

Additionally, changesets can validate that every modified package has a corresponding changeset file using npx @changesets/cli status. This is particularly valuable for monorepos with multiple packages, as it guarantees that all changes are accounted for. To enforce this, we've integrated a CI/CD pipeline that runs on pull requests. Any pull request must include changeset files for modified packages, ensuring that every change is described, justified, and categorized (patch, minor, or major).

By adopting an independent versioning strategy, we ensure that only the packages with actual changes are version-bumped. This approach works seamlessly with changesets, allowing us to accumulate changeset files across multiple pull requests and release them efficiently when ready.

Pre-release Versions

One of the standout features of changesets is its robust support for pre-release versions. By enabling pre-release mode (npx @changesets/cli pre), we can generate pre-release versions of our packages, making it easier to test new features or bug fixes in non-production environments before rolling them out to production. These versions are tagged with a suffix, such as my-package@1.2.3-alpha.0, allowing us to clearly distinguish between pre-release and production-ready packages.

Pre-releases help prevent version pollution. Without them, every test iteration would require a version bump. For instance, fixing a bug in my-package@1.2.3 might take 10 iterations, resulting in versions like my-package@1.2.4, my-package@1.2.5, and so on, up to my-package@1.2.13. This would clutter the version history with incomplete fixes, leading to confusing changelogs and unnecessary noise.

With pre-release versions, we can incrementally test changes using versions like my-package@1.2.4-alpha.0, my-package@1.2.4-alpha.1, and so forth, until the bug is resolved. Once the fix is finalized, we can release the package as my-package@1.2.4, keeping the version history clean and concise. Pre-release versions are stored in a private registry to avoid cluttering the public registry. After the final release, these pre-release versions can be removed from the private registry, ensuring a tidy and streamlined versioning process. This approach provides a much cleaner and more efficient way to manage intermediate changes during development.

Releasing

With a collection of changesets in place, releasing becomes a streamlined process. By running changesets, the changeset markdown files are applied to their respective packages, automatically bumping their versions. For example:

Suppose we have the following packages:

a@1.2.3
b@1.2.3
c@1.2.3

Here, package c depends on b.

Now, consider a single changeset file:

---
"a": minor
"b": major
---
Added a new feature to a, fixed a bug in b.

When we apply the changesets using npx @changesets/cli version, the result is:

a@1.3.0 (minor version bumped due to a new feature)
b@2.0.0 (major version bumped due to a breaking change)
c@1.2.4 (patch version bumped because its dependency b was updated)

changesets handles versioning intelligently, ensuring dependent packages are updated appropriately and eliminating the need for manual version management. This approach guarantees consistent and accurate versioning across all packages.

Additionally, the release pipeline adapts based on the branch. On the main branch, pre-release mode is disabled, and all packages are published as production-ready versions to both private and public registries. On other branches (e.g., staging or dev), pre-release mode is enabled, and packages are published as pre-release versions to private registries only.

Deploying

Deployments can quickly become unmanageable if deployment logic is intertwined with business logic. For instance, in our lambdas package, we have several handlers for serverless functions. These handlers are bundled into a single JavaScript file and deployed to AWS using the aws serverless tool. Previously, the package itself included both the build and deploy commands as npm scripts. However, this approach is unsuitable for our new structure because it tightly couples deployment logic with the package.

If we were to change the deployment logic—such as switching from AWS to Google Cloud—we would need to release a new version of the package to accommodate these changes. Similarly, deploying an older version of the package to a new platform would be impossible because the older version would still contain deployment logic specific to AWS.

To address this, we decoupled deployment logic from the packages and centralized it in a dedicated deploy package. This separation allows us to version our deployment scripts independently. For example:

deploy@1.2.3 contains logic to deploy the lambdas package to AWS.
deploy@1.2.4 contains logic to deploy the lambdas package to Google Cloud.

This setup enables us to deploy any version of the lambdas package to Google Cloud using version 1.2.4 of the deploy package. If we need to deploy an older version of the lambdas package to AWS (e.g., for a legacy hotfix), we can use an earlier version of the deploy package that targets AWS.

This decoupling ensures flexibility, allowing us to adapt to changes in deployment logic, infrastructure providers, or service management without impacting the business logic of the lambdas package.

Another critical improvement we made is ensuring that a package's build artifacts include everything required to run it. This means any version published to a registry can be downloaded and executed without additional dependencies. Runtime data, such as environment-specific configurations, is supplied via environment variables or configuration files. This approach further decouples package logic from deployment-specific configurations, enhancing portability and maintainability.

The deploy package is responsible for preparing the environment by populating variables and configuration files before deploying a specific package version to a target environment. It can be executed using commands like:

npx @prosopo/deploy --package lambdas@1.2.3 --environment staging

This command can be run locally or within CI/CD pipelines, providing flexibility and maintaining development velocity.

CI/CD Pipelines

Here’s an overview of our CI/CD pipelines:

test: Executes automated tests to validate code changes. This pipeline is triggered by pull requests into main or staging and blocks merging if any tests fail.
lint: Ensures code quality by running linting checks. Similar to the test pipeline, it is triggered by pull requests into main or staging and prevents merging if issues are detected.
changesets: Verifies that changeset files are included for all modified packages. This pipeline runs on pull requests to any branch, ensuring that all changes are properly tracked and documented.
release: Processes changeset files to bump package versions. This pipeline is triggered by pull requests into main, staging, or build/* branches. For branches other than main, it generates pre-release versions of the packages.
deploy: A manually triggered pipeline that uses the deploy package to deploy specified packages to a target environment. This pipeline simplifies deployments by automating the execution of the deploy command within the CI/CD system.

You can checkout our workflows for GitHub Actions here.

Example: Fixing a Bug in the `lambdas@1.2.3` Package

Let’s walk through an example of fixing a bug in the lambdas@1.2.3 package using our streamlined process.

Step 1: Create a Bug Fix Branch

Start by branching off main to fix/my-bug-fix. Implement your initial fix for the bug, then run the following command to create a changeset file:

npx @changesets/cli add

This generates a file in the .changesets directory with the following content:

---
"lambdas": patch
---
Fixed a bug where the API would return incorrect data under certain conditions.

Step 2: Trigger a Pre-Release Build

Next, create a build branch from fix/my-bug-fix named build/my-bug-fix and push it to the remote repository. This triggers the release pipeline, which detects the changeset file and recognizes that the branch is not main. As a result, a pre-release version of the package is created: lambdas@1.2.3-my-bug-fix.0. This version is then published to the GitHub NPM registry under private permissions.

Step 3: Deploy to a Development Environment

To test the fix, deploy the pre-release version to a development environment using the following command:

npx @prosopo/deploy --package lambdas@1.2.3-my-bug-fix.0 --environment dev-env-1

This deploys the lambdas@1.2.3-my-bug-fix.0 package to the dev-env-1 environment using the deployment logic in the deploy package, where you can QA the fix.

Step 4: Iterate if Necessary

If the bug persists, return to the fix/my-bug-fix branch, make additional changes, and repeat the process. Each iteration produces a new pre-release version, such as lambdas@1.2.3-my-bug-fix.1. Deploy the updated version to the development environment for further testing:

npx @prosopo/deploy --package lambdas@1.2.3-my-bug-fix.1 --environment dev-env-1

Once the fix is verified, you’re ready to proceed.

Step 5: Merge and Release

Submit a pull request from fix/my-bug-fix to main. The following pipelines will run to ensure quality:

changesets: Verifies the presence of a valid changeset file.
test: Ensures all tests pass.
lint: Checks for code quality issues.

If any pipeline fails, address the issues and repeat the QA process to confirm the fix.

Once the pull request is merged, the release pipeline detects the main branch and performs a full version bump. The changeset file is processed, and the lambdas package is updated to 1.2.4. This version is published to both the GitHub NPM registry and NPMJS registry.

Step 6: Deploy to Production

To deploy the new version to production, run:

npx @prosopo/deploy --package lambdas@1.2.4 --environment production

Alternatively, you can manually trigger the deploy pipeline with the same arguments. This deploys lambdas@1.2.4 to the production environment.

Step 7: Clean Up

After the release, clean up your branches:

Delete the fix/my-bug-fix branch after merging.
Delete the build/my-bug-fix branch after the release.

To simplify this, we’ve configured GitHub to automatically delete branches after merging. Additionally, we have a manually triggered pipeline to purge build/* branches when needed.

By following this process, you can efficiently fix bugs, test changes, and deploy updates without blocking other developers. The use of changesets ensures proper versioning and changelog updates, while the pre-release workflow allows for thorough QA before production deployment.

Why not develop directly on the build branch?

You might wonder why we don’t simply work directly on the build/my-bug-fix branch instead of creating a separate fix/my-bug-fix branch. The reason is that every change pushed to a build/* branch automatically triggers the CI/CD pipeline, initiating a build and release process. Frequent pushes during development would unnecessarily consume CI/CD resources and clutter the registry with multiple development builds. By working on the fix/my-bug-fix branch, we can iterate freely without triggering builds, pushing to the build/my-bug-fix branch only when we’re ready to create a package build for QA or release.

Leveraging Previous Deployment Logic

Suppose you've recently migrated to Google Cloud, replacing the AWS-specific deployment logic in the deploy package. The AWS logic was part of deploy@1.0.0, while deploy@2.0.0 introduced the Google Cloud deployment logic. You deploy lambdas@1.2.4 to production using deploy@2.0.0, but later discover that the Google Cloud is having issues.

To address this, you decide to temporarily revert to the AWS deployment logic while troubleshooting the Google Cloud setup. Thanks to the decoupled deployment logic, this is straightforward. Simply install the older version of the deploy package (deploy@1.0.0) and use it to deploy the lambdas@1.2.4 package to production:

npx @prosopo/deploy@1.0.0 --package lambdas@1.2.4 --environment production

And just like that, the lambdas@1.2.4 package is successfully deployed using the AWS deployment logic. This seamless rollback ensures production remains unaffected while you work on resolving the Google Cloud issues. By decoupling deployment logic from the package, you maintain flexibility and avoid unnecessary downtime during transitions.

Parallel Development Workflow

Here's an example of how developers can work on the same package in parallel using our updated release and deployment pipeline.

Imagine Developer A is fixing a bug in the lambdas package, while Developer B is adding a new feature to the same package. Both developers start by branching off main into fix/my-bug-fix and feature/my-new-feature, respectively.

Each developer creates a changeset file to document their changes—Developer A for the bug fix and Developer B for the new feature.

For simplicity, let’s assume both changes have passed QA and pre-release testing and are ready to be merged.

When Developer A merges their branch into main, the release CI/CD pipeline runs, bumps the version of the lambdas package to 1.2.4, and publishes it to both the GitHub NPM registry and NPMJS registry as lambdas@1.2.4.

Later, when Developer B merges their branch into main, the release CI/CD pipeline runs again, bumps the version of the lambdas package to 1.3.0, and publishes it as lambdas@1.3.0.

This workflow allows both developers to work independently without blocking each other. Changesets ensure that their updates are versioned correctly and consistently, while the release pipeline guarantees timely releases to production. Each change is isolated into its own version, with its own release and corresponding changelog entry.

Key Consideration: Avoiding Race Conditions

To maintain the integrity of the versioning process, the release CI/CD pipeline must not run concurrently. Concurrent runs can create race conditions, leading to mismatched version numbers or incomplete releases.

To prevent this, GitHub Actions allows you to define a concurrency key in the workflow file. This ensures that only one release pipeline runs at a time, queuing subsequent runs until the current one finishes.

concurrency:
	group: ${{ github.ref }}
	cancel-in-progress: false # Enqueue new runs if one is already in progress

This configuration guarantees that the release pipeline processes changes sequentially, preserving the accuracy of versioning and ensuring a smooth development workflow.

Common Pitfalls and How to Avoid Them

Implementing versioning and release pipelines can be challenging. Here are some common pitfalls and how to avoid them:

Forgetting to Create Changesets
- Problem: Developers may forget to create changesets for their changes, leading to untracked modifications and inconsistent versioning.
- Solution: Enforce changeset creation by adding a CICD pipeline that checks for changeset files for any modified packages. This ensures that pull requests cannot be merged without proper changesets.
Mismanaging Pre-Releases
- Problem: Pre-releases can pollute version history if not handled correctly, leading to confusion and unnecessary version bumps.
- Solution: Use pre-release versions (alpha, beta, etc.) for testing and ensure they are published to private registries. Clean up pre-release versions after the final release to keep the registry tidy.
Skipping QA
- Problem: Rushing changes directly to production without proper QA can lead to bugs and broken features.
- Solution: Always deploy pre-release versions to staging or dev environments for QA before merging into main. Automate this process with CICD pipelines to ensure consistency.
Concurrent Release Pipeline Runs
- Problem: Running the release pipeline concurrently can cause race conditions, leading to incorrect versioning or failed deployments.
- Solution: Use the concurrency key in GitHub Actions to ensure only one release pipeline runs at a time.
Tightly Coupled Deployment Logic
- Problem: Embedding deployment logic within packages makes it difficult to adapt to infrastructure changes or redeploy older versions.
- Solution: Decouple deployment logic into a separate deploy package, allowing flexibility in deployment strategies.

Rollback Strategies

Even with a robust pipeline, bad releases can happen. Here's how to handle rollbacks effectively:

Identify the Problem
- Monitor production environments for issues after a release. Use logs, metrics, and user feedback to identify the root cause.
Redeploy a Previous Version
- Use the deploy package to redeploy a stable version. For example:
```
npx @prosopo/deploy --package lambdas@1.2.3 --environment production
```
- This ensures that the problematic version is replaced with a known good version.
Fix and Test
- Create a new branch to fix the issue. Use pre-release versions to test the fix in a staging or dev environment before releasing it to production.
Document the Incident
- Record what went wrong, how it was fixed, and how to prevent similar issues in the future. This helps improve processes and avoid repeated mistakes.
Automate Rollbacks
- Consider adding a rollback script or pipeline to automate the redeployment of previous versions. This can save time during critical incidents.

Conclusion

By implementing a streamlined release and deployment pipeline using Changesets, we’ve significantly improved our development workflow. This approach allows us to manage versioning, releases, and deployments efficiently, reducing friction and increasing developer velocity. We’ve also decoupled deployment logic from package code, enabling us to adapt to changes in infrastructure or service providers without impacting our core business logic. This flexibility is crucial for maintaining a scalable and maintainable codebase. As we continue to grow, we’ll keep refining our processes and tools to ensure we can deliver high-quality software at scale. We hope this post provides valuable insights into building a robust CICD pipeline that meets the needs of a growing software team.

Glossary

Changesets: A tool for managing versioning in monorepos. It creates markdown files describing changes to packages (e.g., patch, minor, or major updates) and automates version bumps and changelogs.
Pre-Release Versions: Versions tagged with a suffix (e.g., 1.2.3-alpha.0) used for testing changes before releasing them to production. These versions are typically published to private registries.
Independent Versioning: A versioning strategy where each package in a monorepo is versioned independently. Only packages with changes are bumped, reducing unnecessary updates.
Locked Versioning: A versioning strategy where all packages in a monorepo share the same version number. Any change to one package results in a version bump for all packages, which can lead to inefficiencies in large projects.
CICD: Continuous Integration and Continuous Deployment. A set of practices and tools that automate the process of integrating code changes, running tests, and deploying applications to production.
CICD Pipeline: A series of automated steps that run in a specific order to build, test, and deploy code changes. Each step is defined in a configuration file (e.g., YAML) and can include tasks like running tests, building packages, and deploying to environments.
GitHub Actions: A CI/CD tool integrated into GitHub that allows developers to automate workflows, such as building, testing, and deploying code.
Monorepo: A single repository that contains multiple packages or projects. This approach simplifies dependency management and versioning across related projects.
Multirepo: A software development strategy where each package or project is stored in its own repository. This can lead to versioning challenges and increased complexity in managing dependencies.