A building-controls platform polls each of 300000 controllers over HTTPS every five minutes to read nine values. That is about 1000 requests a second of pure polling and a full TLS handshake on roughly 8% of them. The team wants device-initiated publish over a persistent session instead. Firmware updates reach only about 70% of the fleet in a quarter. Sequence the migration.
Show the full answer Hide the answer
The sequence
Each step is reversible on its own, and the order is chosen so that no device changes behaviour before the server can tolerate both behaviours at once.
- Stand up the publish path server-side only. Broker, authentication against the same device identities, topic scheme, and an ingest consumer writing to the store the poller writes to. Add a
sourcecolumn to every row. No device is touched; reversible by switching the broker off. - Make the write path idempotent on (device, metric, device-reported instant). A value arriving twice - once by poll, once by publish - must collapse to one row. This must land before any device publishes, because it is the only thing that makes a dual-run safe, and retrofitting it after duplicates exist means a backfill.
- Ship firmware that publishes and keeps answering polls, gated by a server-delivered per-device flag that reverses without another release.
- Cut over per device, not per fleet. When the platform has seen twelve consecutive publishes arriving on schedule from a device, stop polling that device. The decision lives in server state, so reversing it is a flag write.
- Shrink the polling tier as the publishing cohort grows, keeping it alive for the long tail with its target recomputed weekly from the version distribution.
- Decommission polling only when the non-publishing cohort is below the number you are willing to write off - said out loud before you start.
Where data can diverge, and how you would know
A poll samples the value at poll time; a publish reports it at change time. The same day's series has different sample points before and after a device's cutover, so every dashboard comparing this week with last crosses that boundary, and a controller whose temperature oscillates looks spikier under publish.
Keep source on every row forever so any analysis can split by it, and alert on per-device time since last publish exceeding three expected intervals, the signal that replaces "the poll failed". A device that stops publishing produces silence, where one that stopped answering polls produced an error. You are trading an active health check for a passive one, so the absence detector is not optional.
The point of no return
Two moments only. Changing the idempotency key after both sources are live, because the de-duplication semantics of already-written data change under you. And decommissioning the polling fleet, after which a device that falls back has nothing to fall back to. Everything before those is a flag.
The rollback at each stage
Stages 1 and 2: turn the broker off; the poller never stopped. Stage 3: clear the publish flag; those devices are still polled. Stage 4: set the per-device polling flag back on - worst case five minutes of staleness plus duplicates step 2 collapses. Stage 5: scale the polling tier back up, which requires not having deleted its capacity plan.
How long it really takes
Bounded by firmware, not engineering. At 70% reach per quarter the curve is roughly 70%, 91%, 97%, 99% - four quarters to get within a percent, and the last few thousand devices may never arrive. The engineering is weeks; the migration is a year, and the deciding number is the reach curve.
When this is the wrong answer
If the 1000 requests a second is not actually hurting, do not run a year-long migration. The honest first move is to attack the 8% handshake rate with session resumption and connection reuse - a configuration change rather than an architecture change, which may remove most of the cost that motivated the project. Device-initiated publish earns its migration when you need sub-interval latency on state changes, or when connection cost rather than request cost is the constraint. Polling at a five-minute interval on mains-powered devices is unfashionable and perfectly defensible.