Distributed Workflow Orchestration Platform  ·  View 17 of 31  ·  Runtime

At-Least-Once Delivery & Idempotency

The duplicate the platform promises will happen, and what stops it becoming a duplicate side effect.

Editable source SVG draw.io All views
Service Bus
Service Bus
Worker 1
Worker 1
Worker 2
Worker 2
Idempotency store
Idempotency store
Downstream system
Downstream system
Orchestration Service
Orchestration Service
Cosmos DB
Cosmos DB
1. deliver taskExecutionId 82f3 · attempt 1
1. deliver taskExecutionId 82f3 · attempt 1
2. claim 82f3 · IN_PROGRESS · lease 5 min
2. claim 82f3 · IN_PROGRESS · lease 5 min
3. POST /customers · Idempotency-Key: 82f3
3. POST /customers · Idempotency-Key: 82f3
4. 201 Created
4. 201 Created
5. pod evicted before recording result
5. pod evicted before recording result
6. lock expired · redeliver 82f3
6. lock expired · redeliver 82f3
7. read 82f3 → IN_PROGRESS, lease stale
7. read 82f3 → IN_PROGRESS, lease stale
8. POST /customers · Idempotency-Key: 82f3
8. POST /customers · Idempotency-Key: 82f3
9. 200 OK · same resource, no second customer
9. 200 OK · same resource, no second customer
10. mark COMPLETED · result ref
10. mark COMPLETED · result ref
11. TaskCompleted · complete message
11. TaskCompleted · complete message
12. completion for 82f3
12. completion for 82f3
13. transition RUNNING → COMPLETED (ETag)
13. transition RUNNING → COMPLETED (ETag)
14. already COMPLETED → no-op, message settled
14. already COMPLETED → no-op, message settled
At-Least-Once Delivery — Duplicate Execution Without Duplicate Effect
At-Least-Once Delivery — Duplicate Execution Without Duplicate Effect
The platform guarantees the key, not the absence of duplicates. Where a downstream cannot honour an idempotency key, the handler declares itself non-retryable and the task dead-letters on first failure instead of risking a second side effect.
The platform guarantees the key, not the absence of duplicates. Where a downstream cannot honour an idempotency key, the handler declares itself non-retryable and the task dead-letters on first failure instead of risking a second side effect.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display

The contract

  • At-least-once delivery is stated, not implied. Exactly-once processing is not offered, because it cannot be offered across a queue, a worker and an unrelated downstream system without distributed transactions nobody wants.
  • taskExecutionId is the idempotency key end to end: message id, idempotency record key, and Idempotency-Key header on the downstream call. One identifier, three enforcement points, no translation layer.
  • The platform guarantees the key is stable and travels; it does not guarantee the downstream honours it. That boundary is drawn deliberately, and it is the single most important thing a task author must understand.

Why the claim record is not enough

  • The trace shows the hard case: Worker 1 completes the side effect and dies before recording it. The idempotency record says IN_PROGRESS with a stale lease, which is genuinely ambiguous — the platform cannot know whether the effect happened.
  • Resolution is to retry with the same key and let the downstream deduplicate, which is why the key is on the wire rather than only in the platform's own store.
  • Handlers declare an idempotency mode: natural (the operation is inherently repeatable), keyed (the downstream honours the key), or unsafe. An unsafe handler is marked non-retryable and dead-letters on first failure rather than risking a second effect.

Risks

  • Every unsafe handler converts a transient failure into operator work. The count of unsafe handlers is a metric worth watching, because it is the platform's real reliability ceiling.
  • The idempotency record TTL of 7 days must exceed the longest possible retry window. A workflow with a long paused branch could in principle outlive it, which is a validation rule rather than a hope.
  • Duplicate suppression at the message layer covers 10 minutes only. Beyond that the handler and downstream carry the entire burden.