Architecture Views

When the issuance log and the live endpoint disagree about which certificate is in use, which one is the platform's truth?

Context

The natural design stores what the platform did: order placed, certificate issued, secret written, task complete. It is cheap, immediate, complete, and wrong in a way that only surfaces as an outage. Between the issuer and the socket there is a process that has to load the new certificate, and that process is outside the platform's control — a web server that was never signalled, a container that caches on startup, a load balancer whose listener still references the old ARN, an operator who copied a file. Every one of those is ordinary, none of them errors, and each leaves the issuance log green while the endpoint counts down. The public postmortems are consistent on this point: the certificate almost always existed somewhere before it expired somewhere else.

Decision

The certificate registry is a projection of what endpoints are observed serving, reconciled against what the platform issued. An independent verification plane opens a TLS connection to every known endpoint on a fifteen-minute cadence and records the serial, chain and not_after it actually received. That observation is authoritative: the expiry clock is raised from it, the renewal scheduler reads due work from it, and lifecycle state changes on it.

How it is realised on AWS

Regional Lambda probers sweep endpoints listed in DynamoDB and write to an observed index with its own TTL; a divergence detector compares observed serial against registry state on every sweep and emits an EventBridge event when they differ. The registry's `state` column is writable by the prober's role and by nothing else on the SERVING transition.

Options weighed

• ChosenObservation is authoritative; the registry is a projection: Closes the gap the outage lives in. Costs a probing plane, and makes unreachable endpoints an explicit architectural gap rather than an invisible one.
• RejectedThe issuance log is authoritative; probing is a monitor beside it: This is what most tooling does. The monitor raises a warning that competes with every other warning, and the state machine keeps saying the renewal succeeded — so the escalation ladder never fires.
• RejectedTrust the delivery adapter's acknowledgement: An adapter can confirm it wrote the file. It cannot confirm the process read it, which is precisely the step that fails.

Consequences

What it buys

• A renewal that did not take effect stays due, so the failure is caught by the mechanism already built to catch failures rather than by a separate alert nobody owns.
• Certificates the platform never issued enter the same state machine as the ones it did, because the prober does not care who issued what it sees.
• The headline metric — certificates expiring while serving — becomes measurable rather than merely asserted.

What it costs

• A whole verification plane exists that a belief-based design does not need, with its own deployment, network reach and cost.
• Endpoints the prober cannot reach degrade to belief, and the platform has to say so out loud rather than quietly averaging them into a coverage figure.

Choose differently when

If every TLS terminator in the estate could be made to report its own live certificate attestably — a mesh where the control plane already knows each proxy's loaded secret version — observation could move from an external probe to an inbound attestation, which is cheaper and has better reach. The rule would not change; only who produces the observation.

Why it holds up over time

The rule names no technology. It survives replacing DynamoDB, AWS, ACME and X.509 itself, because it is a statement about which of two sources to believe. It is also the property a future team would most easily discard, by adding a fast path that marks a renewal complete on issuance to reduce probe cost — which is why SERVING is a state only the prober's role can write.

Should the lifecycle model collapse "the platform has a certificate" and "the endpoint is using it" into one state?

Context

Most certificate tooling has a status field with values like pending, active and expired. Active means issued. The model has no vocabulary for the most common real condition — a valid certificate exists, and something is still presenting the previous one — so that condition is unrepresentable and therefore unreportable. A state machine that cannot express a failure cannot escalate it.

Decision

The lifecycle model carries REQUESTED, ISSUED, INSTALLED, SERVING, SUPERSEDED, REVOKED and EXPIRED as distinct states. ISSUED is written by the issuance plane; INSTALLED by a delivery adapter's acknowledgement; SERVING only by an observation. A certificate never reaches SERVING without a probe, and the difference between ISSUED and SERVING is queryable, reportable and alarmable.

How it is realised on AWS

State transitions are recorded as events on the EventBridge lifecycle bus and projected into the registry. The SERVING transition is written by the prober's IAM role; the issuance plane's role has no permission to write it, so the distinction is enforced by access control rather than by discipline.

Options weighed

• ChosenSeven explicit states, SERVING written only by observation: Makes the failure representable, which is the precondition for escalating it. Costs a richer state machine and more transitions to reason about.
• RejectedThree states — pending, active, expired: Simple, conventional, and unable to describe the condition that causes the outage.
• RejectedOne state plus a "verified" boolean: A flag beside a state is a state nobody joins on. Reports are written against the state; the flag drifts into decoration.

Consequences

What it buys

• "How many certificates are issued but not serving" is a single query, and it is a standing platform metric rather than an investigation.
• SUPERSEDED, distinct from EXPIRED, makes it visible when an old certificate is still being presented after a successful renewal.
• The escalation ladder can be driven by state plus remaining validity, with no special-casing.

What it costs

• More states to document, test and explain to every consuming team.
• Endpoints that cannot be probed sit permanently at INSTALLED, which is honest but looks like an unresolved backlog — and is one.

Choose differently when

If probe coverage were to fall well below the estate — say a platform dominated by managed services that expose no probeable surface — the distinction would produce more noise than signal, and INSTALLED would have to become the terminal healthy state for those classes, with the weakness declared per class rather than hidden.

Why it holds up over time

The state names are protocol-agnostic and survive any change of issuance mechanism. The valuable part is the separation of who may write which transition, which is an access-control shape rather than a schema, and stays correct as components are replaced.

Can observation run inside the same control plane that performs issuance?

Context

Co-locating them is obviously cheaper: one deployment, one set of credentials, one on-call. It also means that the outage most likely to cause a certificate problem — the control plane being unavailable — is the same outage that removes the platform's ability to notice one. The period during which the estate is most at risk becomes exactly the period during which the platform is blind, and the blindness is silent, because a prober that is not running produces no findings rather than an alarm.

Decision

Probers are deployed per region, in their own compute, with their own schedule and their own egress path, outside the primary control plane's failure domain. Probe liveness is itself monitored: an absence of observations is treated as a failure signal, not as an absence of problems.

How it is realised on AWS

Lambda-based probers in eu-west-1, eu-central-1 and us-east-1, each triggered by its own EventBridge schedule, writing to a DynamoDB global table. Each region's sweep completeness is a published metric; a region that stops reporting alarms within one cadence.

Options weighed

• ChosenIndependent per-region probers with liveness monitoring: Observation survives a control-plane outage. Costs three deployments and cross-region data movement.
• RejectedProbing as a job inside the control plane: Cheapest, and blind exactly when it matters. A silent absence of findings is indistinguishable from good news.
• RejectedA third-party external monitoring service: Genuinely independent and good at public endpoints, but it cannot reach internal mTLS listeners, which is the larger population. Worth keeping as a corroborating signal on public hostnames.

Consequences

What it buys

• The expiry clock keeps running during a control-plane outage, so the incident does not also consume the platform's visibility.
• Regional probers naturally observe regional differences — a certificate rolled out unevenly is visible as disagreement between regions.
• Probe liveness becomes a first-class signal, which closes the "no findings" ambiguity.

What it costs

• Three deployments to operate and keep in step, and a real possibility of version skew between them.
• Cross-region writes to the observed index, with the latency and cost that implies.

Choose differently when

In a single-region estate with a genuinely independent control-plane availability story, one prober deployment with strong liveness monitoring would be defensible. The independence, not the multiplicity, is what matters.

Why it holds up over time

"Do not let the watcher share the failure domain of the watched" is older than any of this technology and applies to every monitoring decision the platform will make later. The specific realisation will change; the constraint will not.

Where do certificates the platform never issued belong — in the registry, or in a separate compliance report?

Context

Every estate has them: a certificate uploaded by hand during an incident three years ago, a vendor appliance with its own self-signed chain, a marketing microsite somebody stood up on a subdomain. They are not managed, so a platform organised around its own issuance has no row for them. They are also, disproportionately, the ones that expire — precisely because nothing is renewing them. Treating them as a compliance finding puts them in a report with a quarterly cadence, while the outage they cause has a date.

Decision

Network and DNS scanning and Certificate Transparency monitoring write into the same registry as issuance. A discovered certificate the platform did not issue becomes a first-class registry entry in an UNMANAGED state with a required remediation path, an owner and a deadline. The count of unmanaged certificates is a standing platform metric, not an audit artefact.

How it is realised on AWS

A discovery scanner enumerates the estate's network surfaces and public DNS; a CT monitor watches every managed domain. Findings are reconciled against the registry and published as an unmanaged-coverage metric alongside expiry. CT alerting fires on any certificate issued for a managed domain by a CA the platform did not use.

Options weighed

• ChosenDiscovery writes into the registry as UNMANAGED: One inventory, one clock, one escalation ladder. Costs a permanently visible backlog nobody enjoys looking at.
• RejectedDiscovery produces a separate compliance report: Moves the riskiest population out of the mechanism designed to protect it, on a slower cadence than the risk.
• RejectedDo not discover; manage only what the platform issued: Defines the problem away. The estate does not agree, and the postmortem will be about a certificate nobody had a row for.

Consequences

What it buys

• The riskiest certificates are tracked by the same expiry clock as the safest ones.
• CT monitoring gives a real detection path for mis-issuance against a customer's domain by a CA the platform never used.
• Coverage becomes an honest number: managed as a fraction of discovered, not of registered.

What it costs

• A standing backlog that will be uncomfortable for a long time, and a metric that looks worse the better the discovery gets.
• Scanning an estate produces false positives and touches surfaces other teams own, which needs a social agreement as much as a technical one.

Choose differently when

In an estate where every TLS surface is provably provisioned through one paved road with no exceptions, discovery would be redundant. No estate of this size has ever been that estate, but a greenfield one might be for its first two years.

Why it holds up over time

The principle — the inventory is of what exists, not of what you created — outlives any scanning technology. As CT coverage broadens and service meshes report their own inventories, the discovery mechanism gets cheaper while the decision stays the same.

Are public-trust certificates for customer domains and private-trust workload identities one system or two?

Context

They share almost nothing operationally. Public trust means ACME, an external CA with rate limits, domain control validation, Certificate Transparency, 90-day lifetimes and deadlines set by a standards body. Private trust means an authority the platform owns, attestation instead of validation, 24-hour lifetimes and no external clock at all. The failure modes, the latency budgets and the people who care are all different. Two systems would keep those differences clean. But both populations expire, both need an owner, both need an escalation ladder, and an organisation that runs two certificate inventories has, in practice, two incomplete ones.

Decision

One control plane, one registry, one escalation ladder and one verification plane span both trust domains. The issuance planes beneath them are separate: different protocols, different authorities, different validation, different lifetimes, and no shared code path for signing. What is shared is inventory, escalation and truth.

How it is realised on AWS

A single certificate registry keyed on issuer-and-serial with a `trust_domain` attribute on the managed identity; two distinct issuance paths — an ACME client with two CA accounts, and AWS Private CA behind attestation — both emitting to the same EventBridge lifecycle bus.

Options weighed

• ChosenShared inventory and escalation, separate issuance planes: One answer to "what is closest to expiry". Costs a control plane that must abstract two genuinely different things without pretending they are the same.
• RejectedTwo independent services: Cleaner blast radius, and nobody can answer the estate-wide question without joining two systems by hand — which is the state most organisations are already in.
• RejectedOne system all the way down, with a common issuance abstraction: The abstraction leaks immediately: rate limits, CT, attestation and lifetime have no common shape, and forcing one produces a lowest-common-denominator API that serves neither path well.

Consequences

What it buys

• One expiry dashboard covers the estate, which is the question the platform exists to answer.
• The escalation ladder, ownership model and evidence trail are built once and behave identically for both populations.
• The verification plane is trust-domain agnostic by construction: a TLS handshake does not care who signed.

What it costs

• A public-path incident shares a control plane with workload issuance, so blast radius is wider than two services would give.
• The control plane carries branching that two separate services would not need, and that branching is where the subtle bugs will live.

Choose differently when

If the public-trust population were to become a genuinely separate product — a customer-facing certificate service with its own SLA and its own team — the shared control plane would become a coordination cost rather than a saving, and splitting with a federated registry would be right.

Why it holds up over time

The dividing line chosen here — share the inventory, separate the mechanics — is the one that survives new trust domains. A future third population (device certificates, a post-quantum hierarchy running in parallel) joins the registry and brings its own issuance plane, without renegotiating the architecture.

How many tiers should the private hierarchy have, and how many issuing authorities should be live at once?

Context

A two-tier hierarchy — root signing leaves directly — means the root key is online and in constant use, so a compromise is unrecoverable without replacing the trust anchor in every relying party. A single issuing intermediate is the common compromise, and it produces a single point of failure with a long tail: replacing it means distributing a new authority to the entire estate under time pressure, which is the operation most likely to go wrong. Meanwhile the root itself has an expiry date, typically years out, which is exactly long enough for everyone who understood it to have moved on.

Decision

Three tiers: an offline root used only in a witnessed ceremony, two independent issuing intermediates live at all times, and short-lived leaves beneath them. Either intermediate can carry all issuance, so retiring one is a planned operation rather than an emergency. CA certificates are tracked in the same registry, on the same expiry clock, as leaves, and rotated at half of remaining life.

How it is realised on AWS

Root key in CloudHSM, module offline between ceremonies, quorum-controlled. Two AWS Private CA issuing authorities, both present in every distributed trust bundle from the day they are created. `ca_authority` rows carry `not_after` and appear on the expiry dashboard alongside leaves.

Options weighed

• ChosenOffline root, two live issuing intermediates: Replacement is a 4-hour operation because the replacement is already trusted. Costs a second managed authority's standing charge.
• RejectedOffline root, one issuing intermediate: Cheaper, and makes intermediate replacement an estate-wide emergency involving trust distribution under pressure — the worst possible time to discover which relying parties cannot report their bundle version.
• RejectedTwo-tier: root signs leaves: Simplest, and puts the trust anchor's key online in the hot path. A compromise then requires replacing the anchor everywhere, which is the outcome the hierarchy exists to avoid.

Consequences

What it buys

• Losing an issuing intermediate costs a planned switch, not a distribution emergency.
• Rotation can be rehearsed against the real estate because a second authority is genuinely live rather than theoretical.
• Root expiry — the multi-year failure nobody is on shift for — appears on the same dashboard as tomorrow's leaf.

What it costs

• Two managed authorities cost roughly twice one, and that charge is per month regardless of use.
• Chain construction and trust bundle composition are more complex, and every relying party must trust both.

Choose differently when

If issuance volume and blast radius were small enough that an intermediate replacement could genuinely be done in an hour by hand, one intermediate would be defensible. At 3,800 workloads across three regions, it is not.

Why it holds up over time

Hierarchy depth and the two-live-authorities pattern are independent of vendor and of algorithm. When a post-quantum or hybrid algorithm arrives, the same structure carries the migration: a third intermediate goes live under the new algorithm, is distributed, and issuance shifts — which is the rotation this design already rehearses.

How does a tenant prove they own a domain, and how often should they have to prove it again?

Context

The platform issues publicly trusted certificates for domains it does not own, on behalf of customers, tens of thousands of times. Validation is the only thing standing between that capability and mis-issuance. Per-renewal validation is the safest model and the most fragile: at 24,000 domains on 90-day certificates, it means asking customers to act roughly 400 times a day, and every one of those is a support ticket waiting to happen. One-time delegation is frictionless forever and grants the platform standing authority over a customer's namespace indefinitely, including after the customer has left.

Decision

DNS-01 with CNAME delegation is the primary path: the tenant creates one permanent `_acme-challenge` CNAME into a platform-controlled zone, and the platform answers every subsequent challenge itself. HTTP-01 is offered as a fallback with its trade-off stated — it requires traffic to already be arriving, so it cannot complete before cutover. Control is re-proved on a declared cadence rather than trusted indefinitely, delegation health is checked continuously, and a domain that is repointed or has its delegation removed stops renewing and retires on a schedule.

How it is realised on AWS

A delegated Route 53 zone holds the challenge records; the DNS-01 responder publishes TXT records there during an order. A delegation checker resolves the customer's `_acme-challenge` CNAME on a schedule and raises an early failure long before a renewal is due. Domain rows carry `proven_at` and `reprove_due`.

Options weighed

• ChosenOne-time CNAME delegation, plus continuous health checks and periodic re-proof: Frictionless renewal with the standing-authority risk managed rather than ignored. Costs a delegated zone and a contractual conversation about what the delegation grants.
• RejectedValidate on every renewal: Safest in principle. In practice it converts a silent, working system into 400 customer actions a day, and the failure mode is a customer outage caused by the vendor's own safety mechanism.
• RejectedHTTP-01 only: No standing DNS authority, which is genuinely attractive — but it cannot issue before traffic arrives, so every cutover has a window with no certificate.

Consequences

What it buys

• The tenant acts once, and renewal for the life of the domain needs nobody.
• Delegation removal is detected by a health check rather than by a failed renewal months later.
• The platform can issue for a new subdomain of an already-delegated domain without asking again, which matters for product features that mint hostnames.

What it costs

• The platform holds standing authority to issue for a customer's domain, which is a real trust concession that belongs in the contract and in the security review, not in a footnote.
• A churned tenant who never removes the CNAME leaves that authority in place, so retirement has to be driven from the platform side.

Choose differently when

If a regulator or a large customer required per-issuance proof of control, the design supports it — the validation orchestrator is already per-order — at the cost of the frictionless property. The decision would then be per-tenant rather than global.

Why it holds up over time

ACME and DNS-01 are stable and widely implemented, and the delegation pattern is now the industry norm for multi-tenant custom domains. If validation requirements tighten — shorter validation reuse windows are already being discussed — the continuous health check is what makes tightening a configuration change rather than a redesign.

For workload identity, is revocation infrastructure load-bearing, or is a short lifetime the control?

Context

Revocation is weak where it matters most. CRLs grow and go stale; OCSP is soft-fail in most clients, which means an attacker who can block the check has defeated it. Building revocation infrastructure that relying parties genuinely hard-fail on is possible inside a controlled estate and expensive. The alternative is to make certificates expire faster than a compromise can be exploited, which converts a distributed consistency problem into a scheduling one — at the cost of multiplying issuance volume and shortening how long the estate survives without the control plane.

Decision

Private workload leaves live 24 hours and renew at 8 hours remaining. Expiry is the primary containment mechanism for a compromised workload key; CRL and OCSP exist for the private hierarchy but are the exception path, not the design. The estate's survival time without the control plane is therefore 8 hours, and that number is published rather than discovered.

How it is realised on AWS

A 24-hour issuance profile in AWS Private CA, with cert-manager and mesh SDS renewing at one third of life. The resulting steady state is roughly 46,000 issuances a day, ~32 per minute, with a fleet-restart burst of about 20 per second.

Options weighed

• Chosen24-hour leaves; lifetime as the control: Compromise window bounded by construction, and the renewal machinery runs continuously so it is known to work. Costs signing throughput and a short control-plane outage budget.
• Rejected90-day leaves with hard-fail OCSP inside the estate: Far less signing load, and it makes revocation correctness load-bearing across every relying party — including the ones that will silently soft-fail and nobody will notice until an incident.
• RejectedOne-hour leaves: A tighter compromise window and roughly 24× the issuance volume, with control-plane survival falling to about 20 minutes. The signing budget and the outage tolerance both become the binding constraint.

Consequences

What it buys

• A stolen workload key is worthless within a day without any revocation reaching anybody.
• The renewal path is exercised tens of thousands of times a day, so it is never the untested code path during an incident.
• Rotation of an issuing intermediate drains in 24 hours, because that is the longest-lived leaf beneath it.

What it costs

• Signing throughput becomes the binding scalability constraint, and it is not elastic.
• The control plane's tolerable outage is 8 hours, which is a real operational commitment.
• A fleet-wide rollout is 37× steady state in ten minutes, which has to be absorbed without starving unrelated issuance.

Choose differently when

If measured AWS Private CA throughput or cost cannot carry the volume, lengthening the leaf is the dial — and it lengthens the compromise window and the control-plane survival budget in the same move. Those are one decision, not three, and the record should be updated as one.

Why it holds up over time

The industry is moving decisively toward shorter certificates and automated issuance; nothing here fights that current. If maximum lifetimes fall further, this design needs no change, because it already treats renewal as continuous rather than exceptional.

Should the renewal threshold be expressed in days, or as a proportion of the certificate's lifetime?

Context

"Renew 30 days before expiry" is the conventional rule and it was written when certificates lived for two or three years. Maximum public certificate lifetimes have fallen from 39 months to 90 days within a decade and are still falling; proposals for 47-day and shorter maximums are already in flight. A fixed 30-day threshold against a 90-day certificate renews at one third of life, which is sensible. Against a 47-day certificate it renews at two thirds. Against anything shorter it becomes "renew immediately, always", and the policy silently stops meaning what it said.

Decision

Renewal triggers at one third of lifetime remaining, expressed as a fraction, for every certificate in the estate regardless of trust domain. On a 90-day public certificate that is 30 days; on a 24-hour workload leaf it is 8 hours. The escalation ladder is likewise expressed against remaining validity rather than absolute dates.

How it is realised on AWS

The renewal policy in the versioned repository carries a fraction, not a duration. The scheduler computes the due time per certificate from its own validity window, and spreads renewals with jitter so that certificates issued in one batch do not come due in one hour.

Options weighed

• ChosenA fraction of lifetime remaining: Survives every future shortening of certificate lifetimes with no change. Costs a slightly less intuitive policy to explain.
• RejectedA fixed number of days: Intuitive, conventional, and quietly wrong the next time the industry halves certificate lifetimes — with no error to signal that it has stopped working.
• RejectedPer-class thresholds, tuned individually: Maximum flexibility and a configuration surface that will drift, with each class becoming a separate thing to remember when lifetimes change.

Consequences

What it buys

• One policy covers 24-hour leaves and 90-day public certificates with no special cases.
• The retry window is always two thirds of the certificate's life, which is a meaningful budget at every scale.
• The next industry-wide lifetime reduction requires no change to this platform.

What it costs

• "One third remaining" is harder to explain to a service owner than "30 days before", and will be asked about repeatedly.
• Very short lifetimes make the renewal window small in absolute terms, which puts pressure on retry and on issuance availability.

Choose differently when

If a particular class of certificate had a hard external constraint expressed in days — a partner integration requiring a fixed rotation date, for instance — that class would need an absolute override, declared and justified rather than inherited.

Why it holds up over time

This is the single cheapest decision in the record and the one most likely to still be paying off in ten years, precisely because certificate lifetimes will keep falling and nothing here has to change when they do.

When does a failing renewal become somebody's problem — at the first failure, or when expiry is imminent?

Context

A renewal job that fails and retries is, in most systems, a log line. It becomes an alert when a threshold is crossed, and the threshold is usually close to expiry, because alerting earlier is assumed to be noisy. The result is a long silent period during which the system knows it is failing and nobody who could fix it does. The 3 a.m. page at T-0 and the quiet ticket at T-30 are the same information, discovered at two very different costs — and the platform has the information from the first attempt.

Decision

A failed renewal opens an incident at the first failed attempt, with severity driven by remaining validity rather than by attempt count: notify the owner at 30 days, alarm at 21, page at 14, escalate to the platform on-call at 7. Retries continue with jittered backoff across the whole remaining window, so a transient CA outage costs retries rather than an outage. Any managed certificate reaching 7 days remaining is a page-worthy defect regardless of whether anything has broken.

How it is realised on AWS

Every attempt writes an outcome to the attempt ledger; the escalation ladder is an EventBridge rule set evaluating remaining validity against the registry, with owner lookup from the managed identity. An owner is mandatory for a certificate to be managed at all, so there is always somebody to notify.

Options weighed

• ChosenIncident from the first failure, severity from remaining validity: Discovery moves to T-30. Costs early, low-severity noise that has to be kept genuinely actionable or it will be ignored.
• RejectedAlert when expiry is imminent: Quiet, conventional, and it discards a month of warning the system already had.
• RejectedAlert after N consecutive failures: Attempt count is the wrong unit: five failures on a 24-hour leaf is an emergency, and five on a 90-day certificate is a Tuesday.

Consequences

What it buys

• Nearly every renewal failure is fixed in working hours by the person who owns it.
• The escalation path is identical for both trust domains because it is expressed in remaining validity.
• A systemic failure — a CA outage, a broken adapter — shows up as many simultaneous low-severity notifications, which is a recognisable signature.

What it costs

• Low-severity notifications arrive for problems that would often have resolved themselves on the next retry, and that class has to stay small or it trains people to ignore it.
• Every managed certificate needs a real owner, which is an organisational cost before it is a technical one.

Choose differently when

If the first-failure notification proves to be dominated by self-resolving transients, the right adjustment is a short confirmation window — two consecutive failures, or one hour — not a move back to alerting near expiry. The unit stays remaining validity.

Why it holds up over time

Severity as a function of time-to-consequence rather than of failure count is a general alerting principle. It transfers directly to certificate rotation, token expiry, licence renewal and every other deadline the platform will acquire.

Is a renewal complete when the new certificate has been delivered, or when the consuming process is serving it?

Context

Delivery and use are separated by a step nobody owns. A file lands on a volume, a secret is updated, a listener's certificate is replaced — and the process that terminates TLS carries on with what it loaded at startup. Some servers watch for changes; many do not. Some reload on SIGHUP; some need a restart. The platform's delivery adapter can honestly report success while the endpoint is unchanged, and this is the single most common way a correct renewal produces an outage.

Decision

Triggering reload of every consumer is part of the renewal transaction. The platform detects and reports consumers holding a certificate older than the registry's current one, and a targeted probe follows the reload rather than waiting for the next scheduled sweep. Delivery without a reload path is not a supported integration.

How it is realised on AWS

Mesh SDS pushes to the proxy without touching disk, which makes reload implicit. The node agent signals the process it serves. Edge and ACM binding is an API call that replaces the listener's certificate directly. Any integration that cannot offer one of these is refused rather than accommodated with a manual step.

Options weighed

• ChosenReload inside the renewal transaction, with a targeted probe: Closes the gap at the point it opens. Costs a supported-integration list and the refusal of ad-hoc delivery.
• RejectedDeliver and let consumers pick it up: The path of least resistance, and it puts the most failure-prone step outside the platform's responsibility while leaving the platform accountable for the outcome.
• RejectedDeliver, and rely on the next scheduled probe to catch failures: Better than nothing — the truth rule would eventually catch it — but it converts a deterministic step into a detection latency for no benefit.

Consequences

What it buys

• The most common renewal failure is eliminated at the source rather than detected afterwards.
• The list of supported integrations becomes an architectural statement instead of a support policy.
• Targeted probing after reload gives a ≤ 10 minute p95 verification, far faster than the sweep cadence.

What it costs

• Teams with bespoke TLS termination must adopt a supported adapter, which is migration work the platform is imposing.
• Every new adapter carries a reload obligation, which makes adding one more expensive.

Choose differently when

If every consumer in the estate ran a proxy with hot-reload semantics — a fully meshed environment with no direct termination — the reload trigger would be redundant and delivery alone would suffice. The probe would still be needed for certificates the mesh does not carry.

Why it holds up over time

As service meshes and managed load balancers displace process-local TLS termination, this decision gets cheaper to honour rather than harder. The refusal to support ad-hoc delivery is what keeps that trend from being undermined by one exception.

What happens to the estate when the certificate service itself is unavailable?

Context

A platform that sits on the trust path has two tempting postures and one correct one. Fail closed — refusing traffic when the platform cannot confirm a certificate is valid — is the instinct borrowed from authorisation systems, and here it would take down the entire estate to prevent a theoretical staleness. Fail open in the sense of relaxing validation is worse. The correct posture is neither: the platform's unavailability should stop change and change nothing that is already running.

Decision

Fail static. No failure of the certificate service can revoke, shorten or invalidate a certificate already serving traffic. Issuance queues rather than errors. The estate coasts on remaining validity, and the coasting time — 8 hours, set by the 24-hour leaf renewed at one third of life — is published as a first-class operational metric.

How it is realised on AWS

Certificates live in the consuming workload and at the edge, not in a store the platform must be available to serve. The revocation publication path is independent of issuance, so a revocation can still be seen while issuance is down. Survival time is computed from the shortest live certificate lifetime and published on the expiry dashboard.

Options weighed

• ChosenFail static, with survival time published: The estate survives a platform outage. Costs an explicit, defensible commitment to an outage budget measured in hours.
• RejectedFail closed: Turns a control-plane outage into an estate-wide outage. The certificate service becomes the least available thing in the critical path, which inverts its purpose.
• RejectedPre-issue longer-lived fallback certificates: Extends survival, and does it by widening exactly the compromise window ADR-08 spent signing throughput to narrow. Reconsider only if measured availability proves 8 hours insufficient.

Consequences

What it buys

• A platform incident is a change freeze, not an outage.
• The outage budget is a known number that operations can plan against and that appears in the SLO conversation honestly.
• Revocation remains possible during an issuance outage, because the two paths do not share a dependency.

What it costs

• Eight hours is a real commitment: an outage longer than that begins expiring workload leaves, and the recovery is then a mass re-issuance.
• Publishing the survival number invites the question of whether it is long enough, which is uncomfortable and correct.

Choose differently when

If the control plane's measured availability made an 8-hour outage plausible rather than remote, the answer is a regionally independent issuance capability — which widens the key exposure surface — rather than longer certificates. That trade is stated in the requirement as a genuinely open question.

Why it holds up over time

Fail-static is the correct posture for any platform that provisions credentials rather than authorising requests, and the distinction will keep mattering as more of the estate moves to short-lived credentials of other kinds.

How does the platform know it is safe to start issuing from a new authority?

Context

The classic PKI outage is not expiry, it is ordering: a new intermediate or root goes live, leaves are issued beneath it, and some fraction of relying parties has not received the new trust bundle. The failure is immediate, total for the affected population, and hard to diagnose from the issuing side because everything on that side worked. The usual control is a runbook step — distribute first, then switch — and runbook steps get abbreviated under exactly the time pressure that forces a rotation.

Decision

A new authority is added to every trust bundle and confirmed distributed before any certificate is issued from it, enforced mechanically by a rotation gate rather than by procedure. Relying parties report the bundle version they hold; the gate blocks issuance below full coverage. A consumer that cannot report its bundle version blocks the rotation by name and is an architectural gap with an owner, not an inconvenience to be waived.

How it is realised on AWS

Signed versioned bundles are published to the trust plane's distribution path; each relying party reports its held version, and the coverage table drives a rotation-readiness query. The issuance plane refuses to sign under an authority whose bundle coverage is below 100%, and a canary identity is issued and probed before any real leaf.

Options weighed

• ChosenMechanical gate on reported coverage, unreportable parties block: Makes the ordering unskippable. Costs a reporting obligation on every consumer, and a rotation that a single legacy consumer can stall.
• RejectedDistribute first, then switch, by runbook: Works until the rotation that matters, which is always the urgent one.
• RejectedCross-sign the new authority from the old one: A legitimate technique that preserves validation for clients with a stale bundle, at the cost of a more complex chain and a weaker claim about what is trusted. Worth holding as the escape hatch for a rotation that cannot wait for coverage.

Consequences

What it buys

• The ordering failure becomes impossible rather than unlikely.
• Bundle coverage is a continuously known number, so "is it safe to rotate" is a query rather than an argument.
• Consumers that cannot report are surfaced early, in daylight, rather than during a rotation.

What it costs

• One legacy consumer with no reporting path can stall a rotation that has a deadline behind it.
• Every relying party acquires a reporting obligation, which is integration work across the estate.

Choose differently when

Under a genuine emergency rotation with a deadline shorter than the remaining migration, cross-signing is the escape hatch — taken as an explicit, attributed, expiring exception with an approver, never as a flag somebody sets.

Why it holds up over time

The gate is about ordering, not about X.509. The same shape applies to any credential-issuing system where verifiers must learn a new key before issuers start using it, which includes every signing-key rotation the platform will ever do.

When retiring an issuing intermediate, do you re-issue everything beneath it or let it empty naturally?

Context

Three techniques exist. Forced re-issuance is fastest and is the emergency path, but it is also a mass event that puts the whole estate through issuance at once for a planned change. Cross-signing preserves validation for clients with stale bundles and complicates the chain. Natural drain — stop issuing from the old authority, keep it trusted, retire it when its last leaf expires — is the safest and its duration is bounded by the longest leaf lifetime beneath it, which this architecture has already made short.

Decision

Planned intermediate retirement drains: the outgoing authority stays trusted and stops issuing, and retires when its longest-lived leaf expires naturally. Because private leaves live 24 hours, that drain is a day rather than a quarter. Forced re-issuance is reserved for compromise, where the speed is worth the disruption.

How it is realised on AWS

Both intermediates are always in the distributed bundle, so stopping issuance from one is a policy change rather than a distribution event. The overlap window is computed from the longest live leaf under the outgoing authority and published on the rotation dashboard.

Options weighed

• ChosenNatural drain, bounded by leaf lifetime: Safest, and short precisely because ADR-08 made leaves short. Costs an overlap window during which two authorities are live.
• RejectedForced re-issuance of every leaf: Fast, and it makes a routine retirement into an estate-wide event with a mass-issuance burst. Kept as the compromise path.
• RejectedCross-sign the new authority from the old: Right where relying parties pin or cannot be updated. Here the bundle coverage gate already guarantees they can be.

Consequences

What it buys

• Rotation is a boring, rehearsable operation rather than a scheduled outage risk.
• The drain duration is a computed, published number rather than an estimate.
• Short leaf lifetimes pay for themselves a second time here, which is a real argument for ADR-08 beyond compromise containment.

What it costs

• Two authorities are live throughout the overlap, which is a wider signing surface than one.
• Public-trust certificates at 90 days make the equivalent drain three months, so the two trust domains rotate on very different timescales.

Choose differently when

If the estate ever acquires a relying party that pins an intermediate — a mobile client, an embedded device — drain stops being sufficient and cross-signing becomes necessary. That is a reason to know where pinning exists before it is discovered during a rotation.

Why it holds up over time

Drain-over-cutover is the general pattern for retiring any signing key, and it gets easier as credential lifetimes shorten across the industry.

In a bulk revocation event, does the platform revoke first and then replace, or replace first and then revoke?

Context

A CA mandate to revoke within 24 hours is a deadline, not an instruction about ordering. Revoking first is the literal reading and, for a population of live customer-facing certificates, it is a decision to break every affected endpoint for however long re-issuance takes. Replacing first means running slightly past the moment the platform could have revoked, in exchange for continuity. The two orderings are correct in different circumstances, and the difference is whether the key is believed to be in an attacker's hands.

Decision

For a mandated revocation, a CA distrust event or an intermediate retirement, the platform re-issues and verifies serving before revoking. For a key known or strongly suspected to be compromised, the order inverts: revoke immediately and accept the outage. The inversion is an explicit decision with an approver, recorded with its reason, never an improvisation at 2 a.m.

How it is realised on AWS

The bulk path drives a rate-limit-aware order queue across both CA accounts, with the probe sweep confirming the new serial is serving before the revocation record is written and published. A compromise trigger sets a flag on the operation that reverses the ordering and requires two-person approval to proceed either way.

Options weighed

• ChosenRe-issue then revoke, inverted on known compromise: Continuity by default, decisiveness when it is warranted. Costs a slightly longer window during which the old certificate is still valid.
• RejectedAlways revoke first: The literal reading of a mandate, and it converts a compliance event into a self-inflicted customer outage across tens of thousands of domains.
• RejectedAlways re-issue first: Safe for availability and wrong for a live compromise, where every minute the old certificate remains valid is the attacker's.

Consequences

What it buys

• A CA mandate is absorbed without customer impact, which is the difference between a difficult week and a public incident.
• The compromise path is pre-decided, so nobody is weighing continuity against containment under pressure.
• Both orderings are rehearsed, so the inversion is not the untested branch.

What it costs

• The old certificate stays valid for the duration of re-issuance, which is a real if bounded exposure and has to be defensible to an auditor.
• Two-person approval on the bulk path adds latency to an operation with a deadline.

Choose differently when

If a CA mandate ever came with an ordering requirement rather than only a deadline, the default would have to invert, and the platform's job would become minimising the outage rather than avoiding it — which makes the 18-hour bulk capacity even more load-bearing.

Why it holds up over time

The reasoning — availability by default, containment when the threat is live, with the switch decided in advance — transfers to every credential revocation the platform will ever perform.

How fast must the platform be able to replace its entire public-trust population, and how is that claim kept true?

Context

The CA/Browser Forum's revocation deadlines are measured in hours and days, and they apply whether the platform is ready or not. Recent industry events have forced large operators to replace their entire certificate estate on a 24-hour or five-day clock, and the organisations that struggled were not the ones without a plan — they were the ones whose plan had never been run. Capacity here is not a property that emerges; it is a number that must be designed to and then defended against drift.

Decision

Bulk re-issuance of the full public-trust population completes in ≤ 18 hours, sized to finish inside a 24-hour mandate with margin. Two public CA accounts are live and pre-validated so that all new issuance can shift to the second within one hour. The bulk path is rehearsed at least twice a year against a representative subset, and an unrehearsed bulk path is treated as untested.

How it is realised on AWS

A rate-limit-aware bulk order queue tracks each CA's per-registered-domain and per-account limits and queues rather than failing as a limit is approached. Rehearsals run against a fixed representative subset and publish a measured duration that is extrapolated honestly rather than asserted.

Options weighed

• ChosenDesigned capacity, two live CA accounts, twice-yearly rehearsal: The deadline becomes a number the platform can meet and prove. Costs a second CA relationship and two rehearsals a year.
• RejectedOne CA account, bulk path built if needed: Cheaper, and it makes the next CA incident an existential one — the plan is written during the deadline it is supposed to meet.
• RejectedThree or more CA accounts: More headroom, more relationships to keep validated, and diminishing returns once the second account can carry the whole population.

Consequences

What it buys

• A CA distrust event becomes an operational day rather than a crisis.
• Rate-limit behaviour is known from rehearsal instead of discovered under load.
• The measured duration keeps the claim honest as the domain population grows — a rehearsal that slows down is an early warning.

What it costs

• Two CA relationships to maintain, validate and keep warm.
• Rehearsals consume real issuance quota and real engineering time twice a year for an event that may never come.

Choose differently when

If the 18-hour target proves unachievable against real rate limits, the architecture must know before a CA sets the clock — and the answer is more pre-validated accounts or a smaller blast radius per account, not a revised target with no mechanism behind it.

Why it holds up over time

External deadlines have only ever tightened. Designing to a measured, rehearsed capacity rather than to an assumption is the property that survives whatever the next requirement turns out to be.

Should revocation be published by the same components that issue, and to what availability target?

Context

Revocation is consumed by relying parties the platform does not control, on a path the platform cannot observe, in clients that mostly soft-fail. That combination means two things: the publication path must be more available than anything else in the design, because a check that cannot complete is a check that passes; and it must not share a dependency with issuance, because the single most likely reason to need an urgent revocation is a problem in the issuance plane.

Decision

CRL and OCSP are published on a path independent of issuance, held to ≥ 99.99% monthly — a higher tier than issuance itself. Responses are signed with lifetimes that survive a publication outage. A fail posture is declared per relying-party class, and an undeclared class is treated as a defect: a soft-fail check nobody decided to soft-fail is revocation theatre. Propagation is measured by a daily synthetic canary rather than assumed.

How it is realised on AWS

Publication through CloudFront with globally cached, signed responses, sourced from a path that does not traverse the issuance plane. A daily synthetic issue-install-revoke-recheck canary measures end-to-end propagation and holds the ≤ 5 minute p95 target honest.

Options weighed

• ChosenIndependent publication at a higher availability tier, with measured propagation: Revocation survives an issuance outage and the propagation claim is evidence rather than assertion. Costs a separately operated path.
• RejectedPublish from the issuance plane: One fewer thing to run, and it couples revocation to the component most likely to be the reason for revoking.
• RejectedRely on short lifetimes and publish nothing: Defensible for private workload identity, where ADR-08 already makes this the primary mechanism. Not available for public-trust certificates, where publication is a CA-level obligation.

Consequences

What it buys

• An issuance outage cannot prevent a revocation from being seen.
• Propagation time is a measured number with a canary behind it, not a design assumption.
• Declaring a fail posture per class forces an honest conversation about which revocations actually have effect.

What it costs

• A separate publication path to operate and to keep more available than the rest of the platform.
• The honest conversation reveals that for most browser clients the revocation is advisory, which is uncomfortable and true.

Choose differently when

If short-lived certificates ever became universal for public trust as well — which the industry is moving toward — revocation infrastructure would shrink toward the exception path on both sides, and this tier could be relaxed.

Why it holds up over time

The principle that a verification path must be more available than the issuance path it checks applies to every credential system, and it is consistently the part that gets under-provisioned because it is invisible when it works.

Who generates the private key, and does it ever cross the network?

Context

Central generation is convenient: the platform makes the key, packages it with the certificate, and delivers both, which simplifies onboarding and makes recovery possible. It also means key material traverses the network, exists in the platform's memory, likely reaches a store, and is therefore present in backups, in logs if anyone is careless, and in the blast radius of any compromise of the platform. For a platform whose administrative surface is the highest-value target in the estate, that is a large concession for an onboarding convenience.

Decision

Every private key is generated inside the consuming workload; the platform accepts only a CSR and returns only a certificate and chain. No private key material exists anywhere in this platform, which is a property of the issuance protocol rather than an access control on a store. CSR contents beyond the public key, and full validation tokens, are never logged.

How it is realised on AWS

cert-manager and mesh SDS generate keys in the workload and present a CSR. The registry schema has no column for key material, so there is nothing to protect rather than something protected. Key possession is therefore never recoverable by the platform, which is the intended property.

Options weighed

• ChosenWorkload-generated keys, CSR-only issuance: A full compromise of the platform's databases yields no key. Costs the ability to recover a key for a workload that lost one — which is correct, because that certificate should be reissued instead.
• RejectedCentral generation with distribution: Simpler onboarding, and it puts every key in the platform's blast radius, its backups and its restore path.
• RejectedCentral generation with immediate destruction after delivery: Better, and it depends on a deletion path being correct everywhere forever, including in a backup taken between generation and destruction.

Consequences

What it buys

• A backup restore cannot revive a key that should be gone, because no backup ever held one.
• The compromise story for the platform itself is dramatically narrower: an attacker with the registry has public material and metadata.
• Key possession is proof of identity in a way the platform cannot undermine even accidentally.

What it costs

• Every consumer must be able to generate a key and produce a CSR, which excludes some appliances and legacy software.
• There is no recovery path for a lost key; the answer is always re-issuance, which requires the attestation path to be reliable.

Choose differently when

A class of device genuinely incapable of key generation would need a separate, isolated provisioning path with its own record — and that path would be an exception with a named owner, not a relaxation of this rule.

Why it holds up over time

CSR-only issuance is the direction of travel everywhere, reinforced by hardware-backed key storage becoming ubiquitous. The rule will get easier to hold, not harder.

Where do the CA private keys live, and what does it take to use the root?

Context

The root key is the trust anchor for every private certificate in the estate. Its compromise is not recoverable by rotation, because rotation means replacing the anchor in every relying party — the operation this architecture spends a rotation gate and two live intermediates trying to avoid doing under pressure. The root is also used rarely: a handful of times in its life, to sign intermediates. A key that is rarely used and catastrophically valuable should not be reachable from anything that runs continuously.

Decision

Every CA private key is held in FIPS 140-2 Level 3 or equivalent hardware, non-exportable. The root module is physically offline between ceremonies. Two-person control is required for root ceremonies, issuing-intermediate creation, any relaxation of a profile constraint, and any manual issuance outside a profile. Root recovery is a witnessed ceremony from hardware backup with an RTO of 72 hours — the one recovery path with a human in it by design.

How it is realised on AWS

CloudHSM holds signing keys with quorum-controlled access; the root's module is offline between ceremonies, and ceremony records are witnessed, recorded and retained ten years alongside the issuance ledger. Issuing intermediates carry name constraints where the technology permits, bounding what a compromised private intermediate could mint.

Options weighed

• ChosenHardware custody, offline root, two-person ceremony: The trust anchor is not reachable by any running system. Costs a facility, a rehearsed ceremony, and a 72-hour recovery.
• RejectedOnline root in an HSM: Operationally simpler and puts the unrecoverable key on a network. The convenience is worth very little given how rarely the root is used.
• RejectedSoftware-held keys with strict IAM: Cheapest, and it makes key exfiltration a permissions bug away. For a trust anchor, that is not a defensible posture.

Consequences

What it buys

• No running system can reach the root, so no software compromise reaches the trust anchor.
• Name constraints bound what a compromised issuing intermediate could do, which is the realistic compromise rather than the theoretical one.
• Ceremony records give a decade-long, auditable account of every time the anchor was used.

What it costs

• Ceremonies are expensive, slow and need people with specific roles available, which makes intermediate creation a scheduled event.
• A 72-hour root recovery is a long time, and it is accepted deliberately rather than engineered away.

Choose differently when

Nothing plausible flips hardware custody. What could change is where it lives: a regulatory requirement for sovereign key custody would move the modules on-premises without altering any of the reasoning.

Why it holds up over time

Offline roots and quorum ceremonies predate all of this technology and will outlast it. When a post-quantum hierarchy is needed, it will be created by exactly this ceremony, which is an argument for keeping the ceremony rehearsed rather than merely documented.

On what basis is an issuance request authorised, and what happens to the requests that are refused?

Context

The naive issuance API takes a subject and returns a certificate for it. That makes the requester the authority on its own identity, which means any workload that can reach the API can obtain any identity in the trust domain — and mTLS authorisation downstream is then worthless, because the identity it checks can be minted on demand. Separately, the requests that get refused are the most interesting security signal the platform produces, and they are the ones most commonly discarded.

Decision

Identity is derived from a platform-attested credential the workload cannot forge, never from a self-asserted subject in the CSR. Identity is encoded in a structured SAN — trust domain, namespace, service — so that policy can be written against a parseable identity rather than a string convention. Authorisation is policy: which identities may obtain which subjects under which profile. Every denial is appended to the issuance ledger with the requester, the requested subject and the rule that refused it, and the denial rate is a standing security signal.

How it is realised on AWS

IRSA projected service-account tokens for Kubernetes workloads and IMDSv2 identity documents for EC2 are verified by the attestation verifier; OPA bundles built from the policy repository evaluate the request. A workload in one namespace cannot obtain a certificate for a service in another. Relaxing a constraint requires a second approver and is audited immutably.

Options weighed

• ChosenAttested identity, structured SAN, policy authorisation, recorded denials: Makes downstream mTLS authorisation meaningful. Costs an attestation dependency on every issuance path and a policy surface to maintain.
• RejectedAuthenticate the requester, trust the requested subject: Common, simple, and it lets any authenticated workload impersonate any other — which silently voids every authorisation decision made downstream.
• RejectedEncode identity in the common name by convention: Works until a name contains a separator character or a team invents a new convention. Policy written against a string convention is policy written against a bug.

Consequences

What it buys

• mTLS authorisation downstream means something, because identities cannot be minted by the workloads that would benefit.
• The denial log is a genuine detection signal for lateral movement and for misconfiguration alike.
• Changing the rules is itself two-person and audited, so the control cannot be quietly widened.

What it costs

• Every issuance path needs a working attestation mechanism, which constrains where workloads can run.
• The policy surface must be kept current as namespaces and services are created, or it becomes the thing teams route around.

Choose differently when

If the estate adopted a workload identity standard that already carries attestation and authorisation — SPIFFE with a fully trusted node attestor, for example — this policy layer would become a thin adapter over it rather than a component in its own right.

Why it holds up over time

Attested, structured, policy-authorised identity is where the industry has converged and where it is staying. The realisation will change as attestation mechanisms improve; the three properties will not.