Your update server should show its work

There is a support ticket every team that ships software to devices eventually receives. It reads: "why did device X get version Y?"

Twenty years ago the question did not exist, because the answer was degenerate: everyone got the newest thing. You put a tarball on a mirror, apt or yum or a cron job pulled it, and the entire fleet converged on one version at whatever pace DNS and disk speed allowed. Deployment was a copy operation. It was also a detonation: when the new build was bad, it was bad everywhere, simultaneously, and the industry's postmortems from that era read like weather reports about the same recurring storm.

The fix the industry converged on was to make "who gets the update" a function instead of a constant. Google Play shipped staged rollouts in 2013: pick a percentage, watch crash rates, ramp. Feature-flag systems generalized it: eligibility became a predicate over user or device attributes. Kubernetes made label selectors the vocabulary for "which subset of the fleet", and that vocabulary leaked outward until every scheduling, deployment and monitoring tool spoke it. By 2020, serving decisions everywhere had become genuinely conditional: channel, version, percentage bucket, attribute match, artifact availability.

Which is progress, and also the origin of the ticket. Every condition you add to a serving decision is a branch a human will one day need to reconstruct at 2am. The industry made update decisions smart and kept the observability of a copy operation. Most update servers today will tell you WHAT a device received. Almost none will tell you WHY, and none will tell you why a device received nothing, which is the harder and more common question. "It should be getting 2.1.0 and it says up to date" has four or five possible causes, and the debugging method at most shops is rereading the eligibility code while squinting.

The failure mode is drift, not complexity

Here is the subtle part. The complexity itself is manageable; five eligibility rules is not a lot. The thing that actually burns teams is that the decision logic exists in more than one place.

It starts innocently. The device-facing endpoint has the rollout check. Then someone builds an admin preview ("what would this device get?") and reimplements the walk, because the endpoint's version is tangled up with telemetry writes. Then a batch API arrives for orchestrators and gets its own copy, slightly different because it plans a whole wave at once. Then the seed script for staging environments approximates it with something simpler, because it is just test data.

Now there are four opinions about eligibility. They agree at first. Then targeting rules ship, and three of the four get updated. The preview starts lying. And a preview that can lie is worse than no preview at all, because an operator who has been burned by it once will go back to rereading source code forever, and an operator who has not been burned yet will trust it during an incident.

Kubernetes got this right, and it is worth stating what "this" is: the scheduler is the only component that decides placement, and when you ask why a pod is Pending, kubectl describe replays that same scheduler's reasons back to you. Node affinity mismatch, taint not tolerated, insufficient memory. The explanation is trustworthy for a boring structural reason: it is not a second system describing the first, it is the first system narrating itself.

Decisions as data

The pattern that falls out of this is small enough to state completely. Make the decision a pure function, make it return its reasoning as data, and forbid every other implementation.

When we rebuilt serving in Relayer, the whole thing became one function with this shape:

explainDecision(releases, device) -> {
  release,            // what to offer, or null
  downgrade,          // is this a rollback fallback?
  trace: [
    { version: "2.2.0", verdict: "skipped",
      reason: "label-mismatch",
      detail: "device does not match target \"tier\"" },
    { version: "2.1.1", verdict: "skipped",
      reason: "current-rolled-back" },
    { version: "2.1.0", verdict: "chosen",
      detail: "offered as rollback fallback (downgrade)" },
  ],
}

The walk itself is five checks per release, newest first: is it newer than what the device runs, is it published, do the device's labels match the release's targeting, is the device inside the rollout percentage (sha256(deviceId:releaseId) % 100, so cohorts are sticky), and does an artifact exist for its platform. First release to pass all five wins. A skip is never a dead end; the walk continues downward, which is the property that makes partial rollouts and canary targeting safe to use casually.

The discipline is that this function is the only one. The device endpoint calls it and throws away the trace. The batch endpoint calls it five hundred times and throws away the traces. The dashboard preview calls it and renders the trace. The seed script for demo data calls it so that even fake telemetry is decision-accurate. There is no second implementation to drift, so the preview is correct by construction rather than by test coverage. We did not write a single test asserting the preview matches production, because they are the same code path; there is nothing to assert.

The cost of this is honestly close to zero. Returning a trace from logic you already have is an afternoon. The constraint that everything must call one function occasionally forces a refactor when an endpoint wants a shortcut. That is the entire bill.

What it buys is a different debugging experience. The 2am question stops being archaeology: paste the device id, read the walk. Device outside the rollout bucket, bucket 47, needs under 25. Device missing the canary label. No artifact for linux/aarch64, which usually means the CI upload step silently failed, which is a different incident than you thought you were having. Each of these is a five-second read instead of a half-hour reconstruction.

The lesson generalizes past update servers. Anywhere a system decides things about specific entities repeatedly - pricing, routing, moderation, scheduling - the same three moves apply: one implementation of the decision, reasons returned as data, and the explanation surface wired to the deciding code rather than to a description of it. Systems that decide things about your fleet owe you their reasoning, and the only reasoning you can trust is the kind the system cannot help but tell truthfully.