A team proposes routing every internal workflow through an event-streaming platform because it is scalable and decoupled. What questions do you ask and when is a simpler mechanism correct?
Show the full answer Hide the answer
What is being tested
Whether you can price operational complexity, and whether you know the specific workload characteristics that justify a log rather than reciting its benefits.
The questions to ask
1. How many independent consumers does each event have? If the answer is one, and always will be, you have a queue-shaped problem, not a stream-shaped one. The defining benefit of a log is that a new consumer can be added without touching the producer. With one consumer, that benefit is hypothetical.
2. Do you need replay? Rebuilding a downstream from history, or fixing a consumer bug and re-running, is a genuine and hard-to-replicate reason to choose a log. If nobody will ever replay, retention is just storage cost.
3. Does the producer need an answer? If yes, this is request/response. Forcing it through a log produces correlation IDs, reply topics and timeouts — a synchronous call with worse tooling and much worse debugging.
4. What is the actual volume? A database-backed queue comfortably handles volumes that surprise people. If you are at thousands of messages per second rather than hundreds of thousands, the throughput argument is not doing any work.
5. Who operates it, and at 3am? Brokers, partitions, consumer groups, rebalancing, quotas, retention, upgrades, and the specific knowledge of why a consumer group is stuck. That is a standing tax on a platform team, paid whether or not the workflows need it.
6. Can the organisation govern schemas? A published event is an API consumed by systems you cannot enumerate. Without a registry and compatibility rules enforced in CI, a producer team will break unknown consumers, and it will happen in the first year.
When the simpler mechanism is right
| Situation | Better choice |
|---|---|
| One consumer, one purpose | A database table as a queue, or a managed queue |
| Producer needs the result | A synchronous call |
| Work items rather than facts | A job queue with visibility timeouts and a DLQ |
| Transactional coupling with your own data | A database-backed outbox — trivially transactional |
| Fewer than a handful of async workflows | A queue; revisit at ten |
A database-backed queue has an underrated advantage: it participates in your transactions. Writing
the state change and the work item atomically is one INSERT, whereas with an external broker it
requires the outbox pattern to achieve the same thing.
When the log genuinely wins
Multiple independent consumers of the same facts. Replay as a first-class need. Retention and throughput a database cannot serve. Cross-team integration where the point is that producers do not know consumers — this is the LinkedIn case, where bespoke point-to-point integrations were growing faster than the systems that needed them.
That last one is an organisational argument, and it is the strongest. It is also the one that only becomes true above a certain number of teams.
How to answer without being obstructive
Do not refuse; scope. Pick the two or three workflows that genuinely have multiple consumers or need replay, put those on the platform, and leave the rest on simpler mechanisms. This gives the organisation real experience with the operational cost before it becomes load-bearing everywhere, and it leaves an exit.
The failure to avoid is a platform adopted for every workflow in an organisation that had two workflows needing it — the cost is permanent and the benefit is concentrated in a handful of paths.