An interviewer says - we run order fulfilment across six services with events and no coordinator. Last quarter about 0.4% of orders ended in a state nobody noticed for days. Do not tell me to add tracing. What structural change do you make, and who gets paged?

What the interviewer is testing

Whether you understand that choreography distributes the happy path and leaves the process itself unowned. Six teams each own a step and each step works. Nobody owns the statement "this order reached a terminal state", so nothing is ever late, because nothing has a deadline, and no rota is responsible for a thing with no owner. The 0.4% is not a bug in any service. It is the absence of a component.

The second thing tested is whether you reach for observability when asked for structure. Tracing answers "where is this order" once somebody knows to ask, and the failure here is that nobody asked.

The clarifying questions that change the answer

• Is the terminal state defined in code anywhere, or only in people's heads? If only in heads, there is nothing to alert against and that is the first deliverable.
• Does any step have a deadline? A step without one cannot be detected as stuck, only as absent, and absence is not an event.
• How many of the six services belong to other organisations? A boundary you cannot change decides whether the coordinator owns sequencing or only observes it.
• What is 0.4% in orders a day? At 20,000 a day that is 80 stuck orders daily, a full-time support job. At 200 a day it is a weekly report, and the report may be the right answer.

A strong answer's arc

Make the process a first-class thing with three attributes: a state machine with a named terminal state, a deadline per step, and one owning team.

Introduce a process manager that owns sequencing and timeouts and nothing else - no pricing rules, no inventory logic - and keep each service's internal work choreographed. The detection mechanism is the per-step deadline: a step that has not produced its completion event inside its deadline raises an alert naming the order and the step. That is structural rather than monitoring, because the deadline is a property of a process definition that did not previously exist.

Then introduce it without taking control first. Run the state machine in shadow for two weeks: consume the events that already flow, assert the expected sequence, alert on violations, drive nothing. You learn the real distribution of stuck states before the coordinator becomes a dependency, and that data is what justifies the deadlines you pick.

On paging: the team that owns the process manager, and it must own a business outcome. If the only candidate is the platform team, the finding to report is that this process has no business owner, and no architecture fixes that.

Common weak answers

• "Add distributed tracing." Visibility without ownership. Tracing turns a four-hour investigation into a four-minute one and does not start the investigation.
• "Add a dead-letter queue." It catches messages that failed. This process stops because an expected event never arrived, and the absence of a message is not a message, so there is nothing to dead-letter.
• "Move everything to orchestration." Pays coordination cost on the five steps that were fine and makes the coordinator a release dependency for six teams. The problem is one missing capability, not the style.
• "Add a reconciliation sweep." Defensible as a stopgap and often the right thing to ship in week one. By design it finds stuck orders hours or days late, so it is a floor rather than a fix.

What a strong answer adds

The boundary rule that generalises it: orchestrate where you need an answer about the whole, choreograph where reacting is enough. In practice, orchestration across ownership and trust boundaries and choreography inside one.

And the cost. A process manager is a new stateful service with its own availability requirement, and it is now down when fulfilment is down. The deadlines will be wrong at first and will generate noise, which is the argument for the shadow period.