Every red box is a failure with its declared behaviour underneath. Declared is the operative word: 'what happens when Service Bus is unavailable' is an architectural answer, not something to discover during an incident.
The execution plane is expected to fail constantly and is designed for it. The control plane must fail safe. Dependencies must have stated behaviour. The last box is what was consciously not solved.
Each row has a chaos experiment in staging that reproduces it and a runbook that resolves it. A failure mode without a test is a hypothesis.
Declared dependency behaviour
Cosmos throttled: SDK retry with backoff, autoscale absorbs, and submissions shed with 503 before orchestration is starved. Reads degrade before writes.
Service Bus unavailable: submissions are still accepted and committed, outbox rows accumulate, dispatch resumes on recovery with a visible lag metric. Nothing is lost, everything is late.
Redis unavailable: concurrency leases fail open at a globally reduced cap and circuits fail permissive. Liveness is preserved, exact fairness is not — a deliberate trade.
Accepted residual risk
Single write region: RTO 4 hours, RPO 5 minutes. Multi-region write was rejected because conflict resolution on a state machine is a correctness problem, not a merge problem.
In-flight messages are not replicated and are rebuilt from Cosmos on failover. Exactly-once is not offered at all; at-least-once plus idempotency is the contract.
A 500-task execution concentrates on one logical partition, and the control plane is shared across all tenants. These are the two items a reviewer should push hardest on, and they are the first two things V2 should address.
