A self-service database provisioning workflow takes 40 minutes and fails about 15% of the time. The platform team says the cloud provider's API is unreliable. Retrying the whole workflow usually works. What is actually going on, and what would you change?

The first three things to look at, in order

1. Where in the 40 minutes the failures land. Clustered at one step is a bug in that step; spread across steps is a systemic property such as a shared quota or a credential expiry.
2. What is left behind after a failure. Run the workflow, kill it mid-way, and inventory the cloud account. If resources remain, the workflow is not transactional and every failure is also a leak.
3. The exact error bodies, not the workflow's summarised status. "Provider error" hides three different causes that need three different fixes.

The diagnosis

"The cloud API is unreliable" is almost always three separate things, and each is fixable.

• Eventual consistency in the provider's control plane. A role, key or subnet is created and is not yet visible to the next call, which returns not-found or access-denied. The naive workflow treats this as a hard error. The correct handling is to poll for readiness rather than to assume create-then-use works, and it is the single most common cause of flaky provisioning.
• Quota and rate limits. Concurrent provisioning from several teams hits an account-level limit, so the failure rate rises with adoption. This is the one that gets worse exactly as the platform succeeds, and it correlates with time of day rather than with any step.
• Non-idempotent steps. A retry of the whole workflow re-creates resources that already exist, or fails because a name is taken. The reason the full retry "usually works" is often that the leaked resources from attempt one are quietly paying for attempt two's success.

The misleading signal

That a retry works is read as evidence of transient provider faults. It is equally consistent with a race the retry happens to lose less often, and with a leak that makes the second attempt easier. The distinguishing test is cheap: run the workflow twice against the same inputs and compare the resulting resource inventory. If they differ, the problem is yours.

The fix, in order

1. Make every step idempotent and keyed, so a retry converges instead of duplicating. Use a deterministic name or tag derived from the request id.
2. Retry at the step, not at the workflow. A 40-minute workflow that restarts from zero burns time and quota, and it multiplies the leak.
3. Poll for readiness with a deadline after every resource creation that a later step depends on, rather than sequencing on the create call's return.
4. Reconcile. A scheduled job that compares intended state with actual state and cleans up or reports drift turns leaks into a number instead of a surprise on the invoice.
5. Then argue about the 40 minutes. Most of it is usually waiting for one managed resource, and the user-visible fix is to return a usable handle early rather than to make the cloud faster.

The alert that would have caught it earlier

Orphaned-resource count by workflow, not provisioning success rate. Success rate says the system is 85% good; orphan count says each failure costs money forever, which is the number that gets the work prioritised.

When not to fix this at all

If the workflow runs twice a month, 15% failure is three failures a year and a human retry is a perfectly good answer. The economics change with frequency: automate the cleanup when the leak is continuous, and accept manual recovery while it is rare.