An event-collection platform routes customer telemetry to hundreds of downstream destinations with different reliability. What delivery guarantees are achievable, and how should destination failures be isolated?

The achievable guarantee

At-least-once to each destination, independently. Not exactly-once, and not a single guarantee across all destinations — because a destination that is down cannot be made to have received something.

The crucial design decision is that destinations must be independent: one failing destination must not delay, block or lose delivery to the others. That means per-destination queues, per-destination retry state and per-destination checkpoints, rather than one pipeline that fans out synchronously.

Why synchronous fan-out fails

Delivering to twenty destinations in one operation means the slowest determines the latency and any failure forces a choice between failing the whole delivery or silently skipping one. Neither is acceptable, and the common compromise — retrying the whole set — redelivers to destinations that already succeeded, multiplying duplicates.

Fan out by writing to per-destination queues, and let each drain independently.

The controls each destination needs

• Its own concurrency limit, so a slow destination cannot consume shared workers.
• Its own retry policy and budget, derived from its observed behaviour rather than from a global default.
• Its own circuit breaker, tripping on latency as well as errors.
• A dead-letter path with an owner, because messages that cannot be delivered must go somewhere a human looks. A dead-letter queue nobody monitors is a data-loss mechanism with extra steps.
• Replay from a checkpoint, so a destination fixed after a day of failure can be caught up without reprocessing everything.

The transformation problem

Customers define their own transformations and filters per destination, which means arbitrary customer code in the delivery path. That needs sandboxing, a hard time limit, memory bounds, and a defined behaviour when it fails — drop, dead-letter, or deliver untransformed — chosen by the customer rather than by the platform.

The guarantee to state clearly

Consumers must be told the delivery is at-least-once and must deduplicate. Providing a stable message ID is what makes that possible, and omitting it is what makes the guarantee unusable — the consumer has no way to recognise a duplicate even if they want to.