intermediate 2 min answer

Two engineers disagree about what happens when a customer cancels an order, and both have been on the team for years. Rather than settle it in the meeting, what do you do, and what will you probably find?

tracingsequence diagramdocumentationdriftevidence
Show the full answer Hide the answer

The first move

Stop drawing from memory and generate the diagram from a real trace. Take ten cancellations from production, pull their distributed traces, and lay the spans out in time order. The argument ends in twenty minutes, and it ends with evidence rather than seniority.

If there are no traces, the second-best source is the actual logs of one request correlated by id, and the third is reading the code paths together. All three beat the meeting, because both engineers are describing a system as it was when they last worked on it.

What you will probably find

Four things show up so often that they are worth expecting.

  • A hop nobody mentioned. A validation service, an audit write, a feature-flag lookup on the critical path that was added by a different team a year ago.
  • A retry that changes the semantics. The diagram in everyone's head has one call to the payment provider; the trace shows two, 900 ms apart, because a timeout fired and the second one succeeded. Whether the refund is therefore issued twice is now a real question.
  • A synchronous call where everybody believed there was a queue. This is the single most common discrepancy, and it usually happened during an incident, as a temporary measure.
  • An N+1 pattern: one call per order line, invisible in the code because it is in a loop inside a helper.

Why this matters beyond winning the argument

The disagreement was evidence of drift, and drift is the real finding. Two people who know the system well disagreeing means the system changed in a way that never reached either of them. Fixing this instance is small; the useful output is asking why nobody knew, which is usually that the change was made by another team without a shared review, or that the diagram exists but is a year old.

What to do with the diagram afterwards

Commit it next to the code and regenerate it, rather than maintaining it by hand. A sequence diagram of a key flow, produced from a trace and refreshed on a schedule, stays true; one drawn in a diagram tool becomes wrong within a quarter and is then worse than absent, because people trust it. Mark it with the date and the trace id it came from.

When not to bother

For a flow with two hops, or one that is about to be replaced. The cost of this practice is real, and it is justified for the handful of flows where being wrong is expensive: payment, cancellation, account deletion, anything with compensations. Pick three flows, not thirty.