Destination Isolation

Delivering one event to many destinations synchronously means the slowest determines the latency, and any single failure forces a choice between failing the whole delivery or silently skipping one destination. The usual compromise — retrying the whole set — redelivers to destinations that already succeeded, multiplying duplicates.

Isolation resolves it: fan out by writing to a queue per destination, and let each drain independently with its own state.

Why it matters

It changes the achievable guarantee from one fragile promise across all destinations to at-least-once to each destination separately, which is both honest and implementable. A destination that is down cannot be made to have received something; the only question is whether its failure affects anyone else.

Implementation patterns

Each destination gets its own:

• Concurrency limit, so a slow one cannot consume shared workers.
• Retry policy and budget, derived from that destination's observed behaviour rather than from a global default. Heterogeneous destinations cannot share a policy.
• Circuit breaker, tripping on latency as well as error rate, because a slow destination consumes resources for the full timeout and returns nothing.
• Checkpoint, so a destination repaired after a day of failure can be caught up from where it stopped rather than by reprocessing everything.
• Dead-letter path with a named owner. A dead-letter queue nobody monitors is a data-loss mechanism with extra steps.

And the shared requirements: a stable message ID so each destination can deduplicate independently, and an explicitly documented at-least-once contract.

Industry example

Event-routing platforms such as Segment deliver customer telemetry to hundreds of downstream tools whose reliability spans from enterprise-grade to a webhook someone set up once. The architecture's entire value is that a customer's analytics tool being down does not affect delivery to their warehouse.

The additional wrinkle in that domain is customer-defined transformations, which put arbitrary customer code in the delivery path — requiring sandboxing, a hard time limit, memory bounds, and a defined failure behaviour (drop, dead-letter, or deliver untransformed) chosen by the customer rather than by the platform.

Failure scenarios

• Synchronous fan-out, coupling every destination's availability.
• Whole-set retry, duplicating to destinations that already succeeded.
• One global retry policy, wrong for nearly every destination.
• Dead letters with no owner, accumulating silently.
• No stable message ID, leaving consumers unable to deduplicate.
• Unsandboxed customer transformations, so one customer's bad code affects the pipeline.

Trade-offs

Per-destination queues multiply infrastructure: a queue, a worker pool, a set of metrics and an alerting surface for every destination. At hundreds of destinations per customer and many customers, that is a substantial operational and cost burden, and it typically forces a shared-infrastructure implementation with logical rather than physical isolation — which is weaker and must be understood as such.

The alternative is coupling, and coupling in a fan-out system means the least reliable destination determines the platform's reliability. That is not a trade any customer would accept if it were stated plainly.

Interview question

"One customer's webhook destination starts taking sixty seconds and eventually times out on every request. Describe the blast radius in a synchronous fan-out design and in an isolated one, and tell me what the customer's other destinations experience in each."