Provisioning Reconciliation

A self-service provisioning workflow is a distributed transaction against APIs that do not offer transactions. It creates a role, a network, a database, a secret and a DNS record, and any step can fail after the previous ones have succeeded. Without a reconciliation pass, each failure leaves resources behind that nobody is tracking and everybody is paying for.

The usual diagnosis, "the cloud provider's API is unreliable", is almost always three fixable things: control-plane eventual consistency where a resource is created and not yet visible to the next call, quota and rate limits that bind harder as adoption grows, and non-idempotent steps that duplicate on retry.

Why it matters

The failure is silent and compounding. A 15% failure rate on a workflow run fifty times a week is several orphaned resource sets a week, each costing money indefinitely, and nobody notices until the cloud bill is questioned, at which point the archaeology is expensive.

There is a security dimension too. Orphaned resources include roles, security groups and network paths that no longer belong to any service. They are unowned, unpatched, and invisible to the inventory that security reviews are based on.

Implementation patterns

• Make each step idempotent and keyed on a deterministic name or tag derived from the request id, so a retry converges rather than duplicating.
• Retry at the step, not at the workflow. Restarting a 40-minute workflow from zero burns time and quota and multiplies the leak.
• Poll for readiness with a deadline after creating anything a later step depends on, rather than sequencing on the create call's return. This is the single most common cause of flaky provisioning.
• Tag every resource with the request id and owner at creation. Reconciliation is only possible if resources can be attributed, and tagging after the fact never happens.
• Run the comparison on a schedule, emitting orphan count by workflow, and either clean up automatically or open a ticket with the resource list attached.

Industry example

This is the model that infrastructure-as-code tools and Kubernetes controllers both converged on: a declaration of intent, an observation of actual state, and a loop that reduces the difference. The Kubernetes controller pattern has made it the default mental model for platform teams since 2015, and the lesson that transfers to home-grown provisioning is that the reconcile loop, not the create path, is what makes the system reliable.

Failure scenarios

• Silent leaks, discovered as a budget variance months later.
• The retry that works for the wrong reason: leaked resources from attempt one make attempt two succeed, which reads as evidence of a transient provider fault.
• Reconciliation that deletes something live, when attribution is wrong or a resource was legitimately created by hand. This is why the first version reports rather than deletes.
• Quota exhaustion from accumulated orphans, where new provisioning fails because old failures are still holding limits.

Trade-offs

A reconcile loop is a second system with its own correctness requirements, and a destructive one if it is wrong. The safe path costs time: report-only for a period, then delete only resources matching an owner tag, with a grace period. That deliberately slows cleanup so the first mistake is recoverable. The alternative, cleaning up manually, works until volume makes it somebody's part-time job.

When not to use it

If the workflow runs twice a month, 15% failure is three failures a year and a human retry is a perfectly good answer. The economics change with frequency: build the loop when leaks are continuous, not when they are occasional. Similarly, if the provisioned resources cost nothing to leave behind — a DNS record, a namespace — the tracking is worth more than the cleanup, and report-only is the permanent answer rather than a first step.

Interview question

Q: Your platform's database provisioning takes 40 minutes and fails 15% of the time. The team says it is the cloud provider. You have one week. What do you do, in order?

What a strong answer covers: classifying the failures by step and reading real error bodies rather than workflow status · running the workflow twice with the same inputs and comparing inventories to prove where the non-idempotency is · fixing readiness polling and step-level retries before anything else · tagging and a report-only reconcile job in week one · the orphaned-resource metric as the thing that gets further work prioritised · and only then addressing the 40 minutes, probably by returning a usable handle early rather than making the cloud faster.

Quick check

Quiz: Why is "a full retry usually works" weak evidence that the provider is at fault? Because it is equally consistent with a race and with leaked resources making the second attempt easier; comparing inventories after two runs distinguishes them.

Flashcard: Which metric gets provisioning reliability funded? — Orphaned resources per workflow, because each one costs money forever, rather than success rate, which sounds survivable at 85%.