Canary rings without the platform team

A percentage rollout answers "how many devices get this?" It cannot answer "WHICH devices?" - and for fleets, which is usually the question that matters. Ship to internal hardware first. Ship to one customer who agreed to be the guinea pig. Keep the new build away from the EU sites until the data-path change is reviewed. A random 10% slice does none of that; randomness is precisely the property you don't want.

How labels won

The interesting history here is that the industry had this problem solved three different ways before one vocabulary won.

Active Directory had organizational units: machines placed in a tree, and policy flowed down the branches. Hierarchies are tidy until reality stops being a tree - a laptop that is both "finance" and "Berlin" and "pilot program" has to live in one branch and be special-cased into the others. Config management went the other way with node classification and facts: arbitrary predicates over machine attributes, maximally expressive, and famously easy to turn into an unreadable pile of nested conditionals that only one person at the company could safely edit.

Kubernetes, in 2014, picked a deliberately boring middle: labels are flat key/value pairs, and selectors match them with equality (and later, set membership). No hierarchy, no expression language. The design docs are explicit that this was a restraint decision - overlapping, non-hierarchical classification without a query language you could hurt yourself with. It was the right call, and you can measure how right by where the vocabulary shows up now: node selectors, service routing, Prometheus scraping, IAM resource tags on every cloud, monitoring filters everywhere. "Label your things, select by label" stopped being a Kubernetes feature and became how infrastructure talks about subsets of itself.

So when we added targeting to Relayer, the design question was mostly already answered by a decade of industry convergence. Devices carry flat labels. A release can require them: { tier: ["canary"], region: ["eu", "uk"] } reads as tier is canary AND region is eu or uk. Any-of within a key, all keys must match, exact strings only, no expression language. The k8s selector subset, minus everything sharp - deliberately.

The semantics that make it safe

Two decisions matter more than the matching syntax, and they are worth stealing for any targeting system.

Unlabeled devices fail closed, and fall through. A device with no tier label does not match a release targeted at tier=canary - it skips that release and continues down to the newest untargeted release it IS eligible for. Both halves matter. Fail-closed means a canary build never leaks to the 96% of the fleet nobody labeled. Fall-through means excluding a device never breaks its update path; it just gets the boring release instead of the exciting one. Targeting that can strand devices is targeting nobody will dare use.

Targeting only broadens once a release has been offered. You can add values or drop constraint keys, never the reverse. Narrowing after the fact would orphan devices that already matched - the same lie as lowering a rollout percentage after the cohort is sticky. Both invariants come from the same principle, which Google Play's forward-only staged rollouts encoded years ago: controls should not pretend to undo distribution that already happened.

Percentage and targeting compose: the rollout percent applies WITHIN the targeted cohort, so "50% of canaries" is a sentence the system understands.

The afternoon version

Concretely, with a fleet reporting into Relayer:

Label the ring. If an orchestrator reports device state, attach labels there:

{ "deviceId": "edge-0142", "version": "2.1.0",
  "platform": "linux", "arch": "aarch64",
  "labels": { "tier": "canary", "siteRef": "blr-dc1" } }

Pull-mode devices can carry labels on the check itself, baked into the URL at provision time:

GET /u/APP_REF/stable/linux/aarch64/2.1.0?l.tier=canary

Target the release. On the release's serving panel, add tier = canary - the editor suggests keys and values it has actually observed in your fleet, so typos announce themselves. Publish at 100% of the cohort, or stage within it.

Then verify without waiting, and this is the step most setups cannot do: run the decision preview for a canary device and a standard one. The canary's trace shows 2.2.0 chosen; the standard device's trace shows 2.2.0 skipped - does not match target "tier", then 2.1.0 chosen. The preview runs the same decision function as the serving endpoint, so what it says is what devices get. When the ring looks healthy, broaden the target or drop it entirely, and the release quietly becomes everyone's.

The pattern held up for Kubernetes at planetary scale and it holds up for a two-person team shipping an agent to three hundred routers: flat labels, dumb matching, fail-closed with fall-through, and forward-only changes. Boring vocabulary, carefully chosen, beats clever machinery - that is the whole trick, and it is very learnable.