advanced 2 min answer

A multi-party booking saga is running in production and support cannot answer where a particular booking is. What is missing from the implementation?

shiprocketsagaobservabilitystatesupport
Show the full answer Hide the answer

What is missing

A queryable current state. A saga implemented as a chain of event reactions has its state distributed across the services that reacted, and there is no single place that knows where a given booking is.

Reconstructing state from an event log during a customer conversation is not a viable support model, and it is the most common practical failure of choreographed sagas.

What the implementation must have

  • A durable state machine per instance, with the current step, the completed steps, the pending step and the outcome. Queryable by the business identifier that support and customers use.
  • A terminal state for every path, including "compensation failed" — which must page a human rather than retry forever, since a stuck compensation is money or goods in an unknown position.
  • Idempotent steps and idempotent compensations, because both are retried.
  • A visible intermediate state in the product. A saga trades atomicity for availability, and the currency of that trade is user-visible intermediate state — "payment taken, shipment not booked" must be a status the interface can express rather than an anomaly.
  • Timeouts per step with a defined escalation, so a step waiting on an external party does not wait indefinitely.
  • Reconciliation against the external parties' records, since they are authoritative for what was actually booked and your view of it is a cache.

The step-ordering rule that reduces the problem

Do the reversible things first and the irreversible things last. Every step moved later is a compensation you may never need to run — and when a compensation genuinely cannot exist, such as a notification that cannot be unsent or a package that has been collected, move the step later or gate it, rather than writing a compensating action that lies about being able to undo.

The choice this argues for

Orchestration over choreography for anything business-visible. The orchestrator holds the state, which is precisely the thing support needs — and the objection that it is a single point of failure is answered by durability rather than by distributing the state where nobody can find it.