A team proposes an event-streaming platform for a background job workflow processing a few hundred thousand jobs a day. What is gained, what is paid for, and when is a database-backed queue the better engineering decision?

What a streaming platform buys

• Replay. Consumers can rewind and reprocess history, which is genuinely transformative for analytics, rebuilding derived state, and recovering from consumer bugs.
• Multiple independent consumers of the same stream at different positions, without coordination.
• Ordering within a partition, and very high throughput per partition.
• Durable retention as a first-class feature, so the log is a source of truth rather than a transport.

What it costs

• An operational system of substantial complexity. Brokers, partitions, replication, consumer group rebalancing, retention policy, schema management, and the specific expertise to debug it at 3 a.m.
• A new failure domain that is not the database. When it is unavailable, work stops, and the team now has two stateful systems to keep healthy instead of one.
• Loss of transactional simplicity. Enqueueing a job is no longer part of the database transaction that created the work, which introduces the dual-write problem and its remedy — an outbox — as permanent extra machinery.
• Rebalancing behaviour that surprises teams: a consumer restart can pause processing across the group, which is invisible until it matters.

When a database-backed queue wins

At a few hundred thousand jobs a day — roughly a handful per second — the workload does not require a streaming platform, and a queue in the primary database offers a decisive advantage:

Enqueue is part of the same transaction as the state change that caused it. If the order commits, the job exists. If the order rolls back, so does the job. No outbox, no dual write, no reconciliation. That single property removes an entire class of bug that streaming platforms require additional machinery to avoid.

It is also debuggable with SQL, backed up with the database, and understood by everyone on the team. Modern relational databases handle this workload comfortably with SELECT ... FOR UPDATE SKIP LOCKED and an index on the visibility column.

What would change the answer

Adopt the streaming platform when a concrete requirement appears, not on projected scale:

• Multiple independent consumers needing the same events at different positions.
• Replay of history as a real operational need, not a hypothetical.
• Throughput genuinely beyond what the database can absorb — which is much higher than most teams assume, and should be measured rather than guessed.
• Cross-team event distribution where the producer must not know its consumers.

The engineering-judgement point

The right question is never "which is more scalable". It is "what is the smallest thing that meets the requirement, and what specific evidence would justify the next step up?" Adopting a streaming platform for a workload a database table handles is a common, expensive, and entirely reversible-in- principle mistake that in practice is never reversed, because by then everything depends on it.