advanced 2 min answer

At 09:14 a document-processing platform of JetBrains' shape starts returning wrong results for about 3% of requests. No errors, no latency change. The stack traces contain three frames of framework code and a lambda. It takes eleven hours to find. The code is a textbook strategy-and-registry design. What made this undiagnosable, and which design decision would you change?

jetbrainsdesign patternsdynamic dispatchregistryobservabilityplugins
Show the full answer Hide the answer

The trigger

Two plugins registered a handler under the same key. The registry took last write wins, so which implementation served a request was decided by class-loading order, and a routine dependency bump changed that order for a subset of deployments.

Why it propagated instead of failing

Nothing errored. The wrong strategy returned a well-formed, plausible answer for a shape of document it was not meant to handle. Every health check passed, every latency percentile was flat, and the only symptom was output that was quietly wrong — which is why it was reported by a customer rather than by monitoring.

Why detection lagged eleven hours

Every indirection removes something from the evidence available at 3 AM.

  • The static call graph says nothing, because dispatch is a runtime lookup.
  • The stack trace says nothing about which implementation ran, because the frame is a synthetic lambda name.
  • Logging inside each strategy does not help either, unless the log line carries which strategy it was — and the version that was running is the one nobody suspected.

The team had full observability of timing and errors, and none of the identity of the code that executed. They could see that the request was served and not by what.

The structural fix versus the tempting local fix

The tempting fix is more logging inside every strategy. It does not work, because you still cannot correlate which ran with which request without an identifier that crosses both.

The structural fix is three cheap changes:

  1. Fail fast on duplicate registration. Last write wins is a silent conflict resolution for a condition that is always a bug. Throwing at startup would have converted eleven hours of debugging into a failed deployment.
  2. Record the resolution. Put the selected implementation's identity on the operation — a span attribute or a structured log field — so the question "which handler ran for this request" is one query.
  3. Assert the resolved registry at startup against a checked-in manifest, so an ordering change is a failing test rather than a production surprise.

The general lesson

A pattern that replaces a static call with a runtime lookup trades compile-time knowledge for flexibility, and you owe the difference back as telemetry. Strategy registries, plugin hosts, dependency-injection containers, dynamic proxies and feature flags all make this trade. Each one is fine. Each one needs the resolution recorded, or the flexibility has been bought with the ability to diagnose.

When this is the wrong lesson

The conclusion is not "avoid the registry". A platform that hosts third-party plugins has no alternative, and a direct call would not be an option at any price. The defects were last-write-wins and unrecorded resolution, both of which cost an afternoon to fix and neither of which required giving up the pattern. Removing indirection that is earning its keep, because an incident was hard, is the over-correction to watch for in the postmortem.