A commerce platform hosts storefronts on hundreds of thousands of merchant-owned custom domains, each needing a valid certificate. What architecture handles certificate issuance, renewal and termination at that scale?
Show the full answer Hide the answer
Why this is hard
A certificate is per-domain, merchants add and remove domains continuously, certificates expire on a fixed schedule, and issuance requires proving control of a domain the platform does not own. At hundreds of thousands of domains, any manual step is impossible and any failure mode that requires human intervention becomes a permanent queue.
The unforgiving property: an expired certificate is a hard failure. The browser shows a security warning, not a degraded page. There is no graceful degradation, so renewal must be reliable in a way that most background jobs do not need to be.
The architecture
1. Automated issuance via ACME, triggered when a merchant points a domain at the platform. Domain control is validated automatically — HTTP validation if traffic already routes to the platform, DNS validation otherwise.
2. Renewal far ahead of expiry. Begin at roughly a third of the certificate's remaining lifetime, so there is a long window in which repeated failures can be retried and escalated. Renewing at the last moment converts any transient issuance problem into an outage.
3. Certificates as data in a central store, replicated to every edge terminating TLS. The edge does not hold the authority; it holds a replicated copy, so adding edge capacity does not require re-issuance.
4. SNI-based selection at termination. The client's TLS handshake carries the requested hostname; the terminator looks up the matching certificate dynamically rather than loading a static configuration. At this scale certificate selection is a database lookup on the hot path, which means it needs a cache and a defined behaviour on miss.
5. Expiry monitoring as a first-class SLI, alerting on certificates approaching expiry without a successful renewal — not merely on renewal job failures. The distinction matters, because the dangerous case is a renewal that has silently stopped being attempted.
The failure modes
Rate limits at the certificate authority. Issuance is rate-limited per domain and per account. A bug causing repeated re-issuance can exhaust the limit and block legitimate issuance for everyone on the platform — a self-inflicted outage with a multi-day recovery. Client-side rate limiting and idempotent issuance are mandatory.
Merchants pointing DNS away and back. Validation fails while the domain points elsewhere. The system must tolerate long-running pending states without either giving up permanently or retrying forever.
Domains that are removed but never cleaned up, accumulating renewal attempts for domains nobody uses and consuming rate limit against real ones.
Validation dependent on the traffic path. HTTP validation requires that traffic already reaches the platform, which is not true for a domain being newly onboarded. Both validation methods are needed, chosen by circumstance.
The certificate authority itself as a dependency. A single CA being unavailable or, worse, mass-revoking certificates is a platform-level event. Multiple CAs with automatic fallback is the mature answer, and it is frequently the missing piece.
The transferable point
At scale, certificate management stops being an operational task and becomes a distributed system with a hard deadline: every object has an expiry, renewal is a workflow that can fail, and the consequence of failure is total for the affected domain. It deserves the same design attention as any other critical pipeline — including its own monitoring, its own capacity planning against external rate limits, and its own disaster recovery.