Skip to content

System breakdown

Safe deployment practices for small teams: releases and rollbacks

A system breakdown of safe releases for a small SaaS team: CI checks, a preview environment per change, staged rollouts, alert lines, one-step rollback, expand and contract migrations, and feature flags.

  • Published
  • Updated
  • 10 min read
  • By Dev Talan

Key takeaways

The short version. Each point is explained in full below.

  1. 01

    Set up release practices in this order: required CI checks and branch protection, error tracking tagged with the release version and a rehearsed rollback, the rule that no release breaks the schema the previous release reads, preview environments, then feature flags and staged percentages.

  2. 02

    Stage a risky change from preview to staff accounts to 5 to 10 percent of users to everyone. At low traffic, use named accounts instead, because 5 percent of 200 daily users is only 10 people.

  3. 03

    Write the alert lines before each release, such as a server error rate more than twice the previous version's for 10 minutes, and let the person releasing roll back to the previous build without asking anyone.

  4. 04

    The expand and contract pattern splits a breaking schema change into separate releases: add the new structure, write to both, backfill, switch reads, and remove the old structure only after you would no longer roll back past the switch.

  5. 05

    Feature flags let a startup deploy code without releasing it to users, but each flag needs an owner, a safe default, server-side evaluation for anything sensitive, and a removal date.

Safe deployment practices for small teams make a bad release small and cheap to undo. Every change passes required CI checks, gets its own preview environment, and reaches users in stages. Alert lines written in advance decide when to stop, the previous build is one step away, and database migrations never break the version you might roll back to.

This breakdown is for a SaaS product with paying customers, a few engineers, and nobody whose full-time job is operations. Most of it is settings in the CI and hosting you already use, plus a few written rules. Incidents still happen, but they should reach few users and end in minutes.

What are safe deployment practices for small teams?

#

They are the release habits that limit how many users a bad change reaches and how long it stays live. Each has a small version a two-person team can run.

PracticeWhat it limitsSmallest version that works
CI checksBroken builds and known regressions reaching mainType checks, lint, critical-path tests, and a build, required before merge
Preview environmentsChanges nobody opened in a browser before mergeA deploy per pull request with seeded data and test-mode keys
Staged rolloutsHow many users see a bad changeStaff accounts first, then a percentage of users behind a flag, then everyone
Alert lines and an error budgetDebate about whether to stop during an incidentTwo or three written numbers per release, and one SLO on the core path
One-step rollbackHow long a bad release stays liveThe host's rollback to the previous build, rehearsed on staging
Expand and contract migrationsSchema changes that make rollback impossibleNo release drops or renames what the previous release reads
Feature flagsRedeploying just to turn a change offA flags table, with an owner and a removal date per flag
Safe deployment practices and the smallest version of each that works

Hexlode, an Open Source image tool from Damn Labs, Pixelact's R&D branch, runs the CI and deploy part of this path in a public repository. Every pull request runs type checks, unit tests on three operating systems, browser tests in three engines, CodeQL, and a container smoke test. Each merge deploys staging, each tagged release deploys production, and every image carries a commit tag, so a rollback redeploys an earlier tag.

What should CI check before a change can merge?

#

CI should block any change that fails the checks you would otherwise run by hand, and finish fast enough that nobody is tempted to skip it. Make the checks required through branch protection on main, so a red build cannot merge on a busy day.

  • Type checks and lint.
  • Unit tests, plus integration tests on the paths that cost money or trust when they break: sign-in, billing, permissions, and the core workflow.
  • A production build, because passing tests do not prove the app builds.
  • Migrations applied to a copy of the current schema, so a broken migration fails in CI before any deploy.
  • A dependency audit for known high-severity vulnerabilities and a secret scan, since a key committed once stays in Git history, as our guide to exposed API keys explains.
  • At least one human review. AI review can take the first pass, and our AI code review workflow shows where a person still decides.

Build the deployable artifact once, in CI, and promote that same build from staging to production. Rebuilding for each environment means production runs a build nobody tested.

Why does every change need its own preview environment?

#

Every change needs its own preview environment so reviewers and product owners can click through it at its own URL before it merges, without running the project locally. A preview catches what tests miss, such as a broken layout or a form that saves the wrong field.

  • One per pull request. Created when the pull request opens, updated on every push, and deleted when it merges or closes.
  • Its own data. A seeded database, or a database branch if your Postgres host offers branching. Never a connection to production.
  • Test-mode credentials only. Sandbox keys for payments, email, and other providers, so a preview cannot charge a card or email a customer.
  • Behind sign-in. Previews carry unreleased features and debug settings, so protect them with your host's access controls.

Previews show what a change looks like. They can't show how it behaves with production traffic and data volumes. A staging environment configured like production checks merged changes together, and a staged rollout then shows how the change behaves with real users.

How do staged rollouts work for a small team?

#

A staged rollout releases a change to a small group first, checks the alert lines, and widens only while they hold. For a small team, use a feature flag whose audience grows in steps, because it needs no extra servers and reverses in seconds.

  1. 01PreviewThe author and a reviewer test the change at its preview URL. It moves on when CI is green and the reviewer approves.
  2. 02Internal usersDeploy with the flag on for staff accounts only. Use the feature for real work for a day, and watch error tracking.
  3. 03A percentage of usersTurn the flag on for 5 to 10 percent of users, or for a few named accounts that agreed to try changes early. Hold through a normal day of traffic, then compare error rate and latency with users still on the old path.
  4. 04EveryoneRaise to 100 percent once the alert lines held at each stage. Keep the flag for a week as a kill switch, then delete it along with the old code path.

At low traffic, choose accounts instead of percentages

Five percent of 200 daily users is 10 people, too few to tell a new error from noise. At low traffic, roll out to named accounts you can talk to, and hold each stage longer.

Flag services usually assign users to a percentage by hashing a stable ID, so a user stays in the same group as the percentage rises. If you build your own, do the same.

Each user lands in a fixed bucket from 0 to 99 for each flag. Raising the percentage from 10 to 25 keeps everyone already included and adds more, and hashing the flag key with the user ID means each flag picks a different set of users.ts
import { createHash } from "node:crypto";

export function isInRollout(flagKey: string, userId: string, percentage: number): boolean {
  const digest = createHash("sha256").update(`${flagKey}:${userId}`).digest();
  return digest.readUInt32BE(0) % 100 < percentage;
}

What alert lines and error budgets should stop a release?

#

A release should stop when it crosses an alert line written before it shipped, and new releases should pause when the service has used up its error budget. Write the numbers down in advance, and let the person releasing roll back without asking anyone. Left until the incident, the decision becomes a debate.

Service level objective (SLO)
A reliability target for a service, such as 99.9% of API requests succeeding over 28 days. Start with one, on the path customers pay for.
Error budget
The failures the SLO allows, which is 100% minus the target. A 99.9% objective allows 1 failed request in 1,000.
Alert line
A threshold for one release, measured against the previous version or the users still on the old path, that triggers a rollback when crossed.

The example error budget policy in Google's SRE workbook halts all changes and releases except P0 issues and security fixes when a service has exceeded its error budget for the preceding four weeks, until it is back within its SLO. A small team can adopt the same rule: when the budget is gone, the next work is reliability.

Starting alert lines for a release, to tune against your own baseline:

  • Server error rate on the new version more than twice the previous version's for 10 minutes.
  • p95 latency on the changed endpoints up by more than a third for 15 minutes.
  • Any rise in failed payments, failed sign-ins, or failed webhook deliveries.
  • A background job queue that keeps growing past its normal peak.

Comparing new with old needs the release version on every error, log, and metric. Set Sentry's release option, or the service.version resource attribute in OpenTelemetry, from the commit SHA in CI. During a flag rollout both groups run the same build, so also tag errors and traces with the flag's value and compare the two groups.

How do you roll back a deployment?

#

Roll back by pointing production at the previous build, which CI already made and you already ran, instead of building a revert commit under pressure. If the change sits behind a flag, turn the flag off first, because that is faster than any redeploy.

Most managed hosts keep earlier builds ready. Vercel's instant rollback points production back at an earlier deployment, Render's rollback redeploys an earlier build without rebuilding it, and on Kubernetes kubectl rollout undo returns a deployment to its previous revision. If you run containers yourself, tag each image with its commit SHA and roll back by deploying the previous tag. Afterwards, revert the commit on main so the next deploy does not bring the bug back. A rollback from the dashboard also stops Vercel and Render from sending new commits to production automatically, so turn that back on once main is fixed.

A rollback replaces code. It does not undo what the bad version already did:

  • Schema changes, and data the new version wrote.
  • Emails, webhooks, and charges already sent to other systems.
  • Jobs queued in a new payload format that the old workers cannot read.
  • Cache entries written in a new shape.

Plan for those before release: migrations the old code can run against, job payloads with a version field, and cache keys that include a version.

Rehearse the rollback before launch

Run the rollback on staging, time it, and write the steps into a runbook. An untried path tends to fail on a missing permission or an expired build. In the same session, restore a database backup to a scratch database, because only a restore proves a backup works.

How do you change the database without breaking rollback?

#

Use the expand and contract pattern: split each breaking schema change into backward-compatible steps across separate releases, so the current and previous versions of the code both work at every point. Danilo Sato's article on parallel change on Martin Fowler's site names the three phases: expand, migrate, and contract. The steps below split the migrate phase into three.

  1. 01ExpandAdd the new column or table, nullable or with a default, and leave the old one alone. The previous release ignores it.
  2. 02Write to bothRelease code that writes the old and the new structure but still reads the old one.
  3. 03BackfillCopy existing rows in small batches from a background job, so no single statement holds locks for long.
  4. 04Switch readsRelease code that reads the new structure. If it misbehaves, rolling back is safe, because the old column is still current.
  5. 05ContractOnce you would no longer roll back past the read switch, make the old column nullable if it is required, release code that stops writing it, then drop it in the release after that.
Renaming users.name, a not null column, to display_name in Postgres. The batched update repeats until it changes no rows. The check constraint added as not valid is then validated without blocking writes, so set not null can skip its full table scan. Dropping not null on name lets the next release stop writing it, and the final drop ships one release after that.sql
alter table users add column display_name text;

update users
set display_name = name
where id in (
  select id from users
  where display_name is null
  limit 1000
);

alter table users
  add constraint users_display_name_not_null
  check (display_name is not null) not valid;
alter table users validate constraint users_display_name_not_null;
alter table users alter column display_name set not null;
alter table users drop constraint users_display_name_not_null;

alter table users alter column name drop not null;

alter table users drop column name;
  • Migrations live in version control and run from CI before the new code deploys, never by hand.
  • No release drops, renames, or narrows a column or table that the previous release still reads.
  • Indexes on large Postgres tables use create index concurrently, which avoids blocking writes but cannot run inside a transaction.
  • Each migration sets a short lock_timeout, so one waiting for a lock fails fast instead of queueing every query on that table behind it.
  • Down migrations help in development. In production, the rollback plan is that the previous code still works against the new schema.

When should a startup use feature flags?

#

Use feature flags when a change is risky, large, or needs a staged rollout, and as kill switches around integrations that can fail. Small, low-risk changes do not need one, because every flag adds a code path to test and later remove.

  • Release flags hide unfinished or risky work, and are deleted within weeks of reaching everyone.
  • Kill switches turn off a feature or an integration, such as a slow third-party API, during an incident. They stay, and they are tested like any other path.
  • Plan and permission settings decide what a customer has paid for or may access. Keep them in your own database and authorization code, not in a release flag tool.

A flags table in your own Postgres database covers on and off switches and staff-only releases. Move to a flag service, such as LaunchDarkly, Flagsmith, or PostHog feature flags, when you need percentage rollouts, targeting, and a history of who changed what.

  • Every flag has an owner and a removal date, recorded where the flag is created.
  • Every flag has a safe default for when the flag service cannot be reached, usually the old behavior.
  • Flags that touch data access or money are evaluated on the server, because a browser can change a flag value it receives.
  • Both paths have tests for as long as the flag exists.

Which release practices should a small team set up first?

#

Start with the practices that shorten how long a bad release stays live, then add the ones that shrink how far it reaches. A sensible order for a team moving from manual deploys to a safe release path:

  1. 01Required CI checks and branch protection on main.
  2. 02Error tracking tagged with the release version, and a rehearsed one-step rollback.
  3. 03The rule that no release breaks the schema the previous release reads.
  4. 04Preview environments for every pull request.
  5. 05Feature flags for risky changes, then staged percentages and a written error budget once traffic is large enough to measure.

Products built fast, with AI coding tools or a small contract team, often reach paying customers while deploys still run from one laptop. That was a sensible trade for speed. In a software rescue, the stabilization sprint scoped from the audit usually adds the first two before any larger fix, because every later change depends on a release you can undo.

What should you do next?

#

Take your last production release and answer three questions in writing: how long a rollback would take, which numbers would have told you to stop, and whether the previous build still runs against today's schema. Then time a real rollback on staging this week. The answers show which practice to set up first.

The technical debt triage checklist helps rank release gaps against the rest of your backlog. For an outside view, a software audit follows one change from commit to production and ranks what to fix.

FAQ

Common questions

Short answers to what readers ask next about this topic.

Usually not at first. Blue-green runs two full production environments and switches traffic between them, and a canary routes a share of requests to new servers. Both need control of the load balancer and extra hosting. A managed host's instant rollback plus percentage rollouts through feature flags gives a small team most of the same protection. Traffic-level canaries start to pay off when a few minutes of traffic is enough to measure.

As often as changes are ready, usually several times a week or more. Small releases are easier to review and roll back, and when every error carries the release version, as set up above, a regression points to the few commits in one release instead of weeks of work. Fixed release days make most sense for mobile apps, where app store review sets the pace.

Deploy when someone who can roll back will watch the release for the next few hours. With required CI, a rehearsed rollback, and flags, a Friday morning release carries the same risk as any other. Avoid releasing just before the team goes offline, and hold risky schema changes and large rollouts for days when the people who know that code are available.

A mobile release cannot be rolled back once users install it, so the safety moves elsewhere. Apple's phased release reaches users with automatic updates over seven days and can be paused, and Google Play's staged rollout lets you set and halt the percentage. Remote flags switch off a broken feature without a new store build. Treat every API change as expand and contract, because old app versions keep calling it for months.

Who can deploy and roll back, the exact commands or buttons for each, where to find the previous build, the alert lines and dashboards to watch, how to turn off each kill switch, and who to tell when a rollback happens. Keep the runbook in the repository next to the code, and update it in the same pull request as any change to the release process.

Usually yes, once you have paying customers. A preview tests one change at a time with seeded data. Staging runs main with production configuration, so it catches problems between merged changes, environment settings, and migrations run against a production-sized, anonymized copy of the data. It is also where you rehearse rollbacks. A very small product can start with previews and add staging before its first large migration.

Related

Related services, guides, and checklists.

Monthly newsletter

Get the next article by email

One short email a month on scoping, security, AI features, and releases, with links to new articles and checklists. Every issue is also published on the site, and every email has an unsubscribe link.

By subscribing, you agree to get the Pixelact newsletter by email. Every email has an unsubscribe link. Read the privacy notice.

Start a project

Tell us what you're building.

Share a few details in a short form, or book a discovery call if you're not sure where to start.

On the discovery call

  1. 01What prompted this now, and where things stand
  2. 02What a good outcome looks like
  3. 03Whether we're a fit, and what a next step could be

45 minutes. No slides, and no long questionnaire.