An order spans payment, inventory and shipping, each in its own service with its own database. How do you keep them consistent?

Rule out the distributed transaction

Two-phase commit gives atomicity and isolation across services, and its costs make it unusable at scale: locks held across a network for the duration, a coordinator that is a single point of failure, and a blocking window where a coordinator failure leaves participants unable to proceed or release. Most managed data stores and message brokers do not support it, which settles the question in practice.

Use a saga, and be precise about what it gives you

A sequence of local transactions, each with a compensating action. Atomicity in the sense that the sequence completes or is compensated, durability from each local commit — and no isolation.

That last point is the design work. Each local transaction commits immediately, so intermediate states are visible: another process can read a state that will later be compensated, or overwrite one mid-saga.

Countermeasures, chosen per saga: a semantic lock — a status field marking the record in progress, so others decline to act on it; commutative updates whose order does not matter; pessimistic ordering, sequencing so the most damaging anomaly cannot occur; and re-read and validate before compensating.

Choose the coordination style

Choreography — each service reacts to the previous one's event. Minimal coupling, and no single place knows what the process is, which makes timeouts, compensation ordering and "where is order 123?" genuinely difficult.

A process manager — a component that subscribes to events, holds each instance's state, and issues commands when a step is due. Participants stay decoupled from each other; the process becomes explicit and observable.

For a business-critical flow with compensation, deadlines and a state customers ask about, the process manager is the right answer. Choreography-everywhere teams tend to reconstruct one badly out of status columns and scheduled jobs.

Get the messaging right, or none of it works

The outbox pattern for publishing: write the business change and the outgoing message in one local transaction, relay afterwards. Without it you have a dual write, and any failure between the two leaves an inconsistency nothing can detect.

Idempotent consumers keyed on message ID, because delivery is at-least-once by construction.

Retries with backoff and a dead-letter path for the steps that fail transiently.

Design compensation honestly

Compensation is not rollback. A refund is not the erasure of a charge — it is a new business event, visible to the customer, sometimes not fully reversible. An email cannot be unsent.

So sequence actions such that anything uncompensatable happens last, and accept that some compensations are business processes rather than database operations.

Make it observable

A saga's state must be queryable: which step, how long, what failed. Support will ask, and so will you at 03:00. Alert on sagas stuck beyond a threshold — the characteristic failure is not a crash but an instance that silently stops advancing.

What a strong answer adds

Questioning the decomposition. If payment, inventory and shipping must be transactionally consistent on every order, that is evidence the boundary may be wrong. Sagas are the right tool when eventual consistency is genuinely acceptable to the business — and confirming that with the business, rather than assuming it, is part of the design.