A rollback is not a delete

Every update system has a rollback button, and most of them are lying to you about what it does.

What operators think it means: "undo the bad release." What it usually means: "stop offering the bad release to devices that do not have it yet." Those are very different promises, and the gap between them is exactly the set of devices that already installed the bad build. They are the reason you rolled back. They are also, in most systems, the population nothing happens to.

How the industry got here

In the mirror-and-package era, rollback genuinely was a delete. You removed the bad package from the repository, restored the previous one, and machines that had already upgraded were someone's ssh problem. Configuration management (CFEngine, Puppet, Chef) softened this by making state convergent: declare the old version, and agents walked themselves back. But convergence assumed the machine was still healthy enough to run an agent, which is precisely what a bad release endangers.

Mobile app stores made rollback WORSE, structurally. Neither Google Play nor the App Store lets a user downgrade; when a developer pulls a release, existing installs keep it forever. The industry answer became an idiom: the re-release. You take the last good build, stamp a higher version number on it, and ship 2.1.2 that is byte-identical to 2.0.0. It works, and every mobile team has done it, but notice what it is: a workaround for version monotonicity that the platform imposes, executed manually, during an incident, by someone whose hands are shaking.

Kubernetes is the counterexample worth studying. kubectl rollout undo is not a delete and not a re-release; it is a redeploy of a previous known state, and the system converges on it the same way it converges on anything. Rollback being a first-class, boring operation rather than an emergency ritual is one of the underrated reasons the k8s deployment model won. The lesson: rollback should be a state transition the system understands, with defined behavior for every device, including the ones already on the pulled build.

The three honest questions

Designing rollback for a device fleet, you owe answers to three questions most systems dodge.

What do stranded devices fall back to? Our answer: the newest still-published release below the pulled one, served as an explicit downgrade. The manifest carries allowDowngrade: true so clients that guard against version regression (electron-updater does, by default) know this one is intentional. Serving-side, this is one exception in the decision walk: normally the walk stops at the device's current version, but if the release matching the current version is rolled back, the walk continues below it. A pulled build should never silently keep its victims.

Client capability matters here, and pretending otherwise is how you get surprised: our CLI applies downgrades natively, electron-updater needs its allowDowngrade flag enabled, and Tauri's updater refuses to move down-version at all, in which case you are back to the re-release idiom, publish a higher-numbered build with the old contents. The system should tell you which case you are in, not leave it as a footnote you discover during the incident.

What does the operator see? Counts, not vibes. The rolled-back release row shows how many devices still report the pulled version and exactly what they will be offered: "61 devices stranded on this pulled build, falling back to 2.1.0 (served as a downgrade)." And critically, the rollback confirmation dialog shows the same numbers BEFORE you commit, because the moment you press that button is the worst possible moment to be doing mental arithmetic about consequences. If no published release exists below the pulled one, the dialog says so in amber: these devices will hold until you publish something. That is an ugly truth, and an ugly truth in the confirm dialog beats a discovery a week later.

Can rollback be undone? No, and this is a feature. A rolled-back release in Relayer can never be re-served; if the build was actually fine, you publish it again as a new version. Play works the same way, staged rollout percentages only move forward, halting and pulling are separate verbs. The reasoning generalizes: any control whose meaning is "this artifact was withdrawn for cause" must be irreversible, because every process downstream of it (incident notes, audit trails, the fallback behavior of thousands of devices) depends on it not quietly flipping back. For the same reason, rollout percentages in our system only increase while a release is published. Cohort bucketing is sticky, so lowering the number never un-ships anything; all it does is make the dashboard misreport reality. A control that cannot do what it appears to do should not exist.

The test

Here is a one-question audit for whatever update system you use today. Take your most recent release, imagine it is bad, and ask: for a device that already installed it, what exactly does the next update check return after I roll back?

If the answer is "nothing, they keep the bad build until something newer ships", you have a stop-the-bleeding button, not a rollback. That might be an acceptable trade for your product. But it should be a decision you made, written down where the on-call can read it, not a default you inherited from a system that never asked itself the question. Rollback is a state transition, not an apology, and the devices already on the bad build are part of the state.