Back to Insights
Cloud & DevOpsYesterdayJustin Pennington

You Don't Need a Release Train. You Need a Rollback Button.

You Don't Need a Release Train. You Need a Rollback Button.

Ask a three-person technical team why they avoid deploying on Thursday afternoon and you'll rarely hear "our testing isn't thorough enough." You'll hear something closer to: "If it breaks, I don't know how long it'll take to put it back." That's not a quality problem. That's a reversibility problem — and it's the single biggest reason lean teams accumulate risk while moving slowly.

Big companies solve this with process: change advisory boards, release windows, sign-off chains. That machinery exists because reversing a change in a large enterprise is genuinely hard, so they front-load the caution. Small and mid-market teams copy the ceremony without the underlying need, and end up with the worst of both worlds — slow releases and no safe way to back out.

The Question That Actually Matters

Before you add another approval step, answer this: if the change we ship today turns out to be wrong, how many minutes until we're back to the state we were in this morning?

If the answer is "minutes," you can ship confidently and often. If the answer is "we'd have to figure it out," no amount of review will save you, because the bugs that reach production are the ones nobody predicted. Review catches the problems you thought to look for. Reversibility catches the rest.

This reframe matters most for teams running integrated systems — an ERP connected to a website, a CRM, a fulfillment partner. The blast radius of a bad change isn't one app. It's every downstream process that trusted the old behavior.

What Makes Undo Possible

Reversibility is built from a handful of unglamorous practices. None of them require a platform team.

  • Deploy artifacts, not edits. If a change ships as a versioned package you can redeploy, rolling back is a command. If it ships as someone editing files on a live server, rolling back is archaeology.
  • Feature flags for behavior changes. A flag is just a switch that turns a new feature on or off without redeploying. Ship the code dark, flip it on for yourself, then for everyone. If it misbehaves, flip it off — no release required.
  • Restore drills, not just backups. A backup you have never restored is a hypothesis. Practice the restore on a copy of production, time it, and write the steps down while they're fresh.
  • One change at a time. Bundling eight fixes into one release means a rollback undoes seven things that were working fine. Small batches make the culprit obvious.

Database Changes Are the Exception

Code rolls back cleanly. Data does not. The moment a change alters your database structure — renaming a column, dropping a field, restructuring a table — you've crossed into territory where "just put it back" can mean losing information customers created in the meantime.

The practical answer is to separate structural changes from behavioral ones. Add the new column first and let both old and new code work against it. Deploy the code that uses it. Only after the new behavior has run clean for a while do you remove the old field. It feels tedious in the moment, and it is the difference between a bad hour and a bad quarter.

This is doubly true inside an ERP, where customizations touch records that accounting, inventory, and fulfillment all depend on. Reversible sequencing isn't bureaucracy there — it's the reason you can keep improving the system after go-live instead of freezing it.

What Lean Teams Should Skip

You do not need a staging environment that perfectly mirrors production — most small teams can't afford to maintain the fidelity, and a staging environment nobody trusts gives false confidence. You do not need a formal change board for a team that fits in one conversation. You do not need a release calendar if you can ship safely on demand.

What you need instead is a short, written path from "something looks wrong" to "we're back to normal." Who notices. Who decides. What command they run. Where the flag lives. Posted somewhere findable at 9pm, not buried in a wiki nobody opens.

Where to Start This Week

Pick your most business-critical system and run the exercise cold. Make a trivial change, deploy it, then roll it back — and time the whole thing. You will learn more in that hour than in a month of planning documents. Most teams discover one of three things: the rollback path doesn't exist, it exists but only one person knows it, or it works fine and the fear was outdated.

All three outcomes are useful. Only one requires work.

If your team is shipping carefully because reversing a mistake feels risky — especially across connected ERP, web, and operational systems — that's a solvable architecture problem, not a discipline problem. We help operators build deployment and integration practices that let small teams move fast without betting the business on every release. Reach out and let's look at what your undo path actually looks like today.