Devices are provisioned on a contract manufacturer's line: each unit boots, calls your cloud provisioning service, receives a unique certificate, and is boxed. At 02:00 on a Saturday your provisioning service becomes unreachable from that factory. The line runs three shifts and cannot stop. What happens next, and what should the design have been?

Second by second, what happens

The first units fail their provisioning step and are routed to a rework bin. Within twenty minutes the bin is the fastest-growing thing on the line. The line supervisor is measured on units shipped, not on units provisioned, and has two options that both damage you: stop the line, which costs real money per hour and burns a relationship, or box unprovisioned units and let someone deal with it later.

In practice the second happens, often with an improvised workaround: a technician finds that units provisioned earlier used a test credential that still works, and flashes that. You now have an unknown number of units in the field sharing one key, and no record of which serial numbers they are. The outage lasted 40 minutes; the consequence lasts the product's life.

Where it amplifies

Manufacturing is a hard real-time system with no queue and no backpressure, which is the opposite of the assumption a cloud API makes. A retry with exponential backoff is not a mitigation when the conveyor keeps moving. And the failure is silent from your side: your dashboards show a drop in provisioning calls, which looks like a quiet shift.

What stops it

Provisioning must not require your cloud to be reachable at the moment of manufacture. The designs that survive:

• Key generation on the device, certificate signing later. The secure element generates its keypair at first boot and emits a certificate signing request. The line stores CSRs to a local file, and boxing proceeds. Your service signs them in a batch when it is reachable. The private key never leaves the part, which is the property you actually wanted.
• Pre-issued credential batches. You hand the manufacturer a sealed batch of certificates bound to a serial range, injected offline. It works without connectivity and it means unissued credentials exist outside your control, so batches are small, short-dated, and reconciled.
• Claim on first use rather than at manufacture. The device leaves the factory with only a birth certificate from the silicon vendor's on-die key, and gets your identity the first time it connects from a customer site. The line has no dependency on you at all.

The decision rule

Choose claim-on-first-use when devices reach a network before they reach a customer, and CSR-batching when they might not. Pre-issued batches are the fallback for lines that cannot run new tooling, accepted with an expiry short enough that a stolen batch dies on its own.

When this is over-engineering

A pilot run of 500 units assembled in-house, where the line is three people and stopping it costs nothing, does not need any of this. The threshold is the first time someone else's production schedule depends on your uptime, which is usually the first contract manufacturer and always before volume.

Common weak answers

• "Add retries and a queue in the provisioning client." The line does not wait. Retries convert a hard failure into a slow one, and the units still pile up.
• "Run the provisioning service in two regions." Useful, and it does not address the factory's own connectivity, which is the link that failed here.
• "Give the manufacturer an API key with permission to provision." Now the compromise of a third party's workstation mints identities in your fleet.