A team proposes Kafka for a workload of a few thousand messages per day. When is Kafka over-engineering, what is the alternative, and when does the answer change?

What Kafka is actually for

Kafka is a durable, ordered, replayable log with independent consumer groups. Its distinguishing properties:

• Retention independent of consumption — messages persist after being read, so a new consumer can replay history and a broken consumer can reprocess.
• Multiple independent consumer groups reading the same stream at their own positions.
• Ordering within a partition, with parallelism across partitions.
• Very high throughput, into millions of messages per second per cluster.

A workload of a few thousand messages per day exercises none of these, and paying their cost buys nothing.

The cost being paid

• Operational weight. A cluster to run, upgrade, monitor and understand — even with KRaft removing the ZooKeeper dependency, and even managed, where the cost becomes financial and cognitive rather than operational.
• Partition and key design decisions that affect ordering and parallelism and are awkward to change later.
• Consumer group management, offset semantics and rebalancing behaviour — a genuine body of knowledge, and the source of most operational surprises.
• The at-least-once contract, which is correct and requires every consumer to deduplicate.
• Debugging complexity: a message not arriving may be a consumer lag issue, a partition assignment issue, a rebalance, a serialisation failure or a poison message, and distinguishing them takes expertise.

The alternative at that volume

A table in the database you already have. Insert rows; a worker polls with SELECT ... FOR UPDATE SKIP LOCKED; mark them done.

This is unfashionable and it is genuinely excellent at low volume: transactional with the business change — which removes the dual-write problem entirely — trivially debuggable with SQL, no new infrastructure, ordering and retry semantics that are obvious, and dead-letter handling that is a status column.

It scales further than people assume — comfortably into thousands of messages per second on ordinary hardware — and the point at which it stops working is well beyond where most systems ever reach.

Between the two sit managed queues, which are cheap, operationally trivial, and appropriate when the database is not the right place but a log is not needed.

When the answer changes

• Multiple independent consumers of the same stream, each at their own position. This is the property that most often genuinely justifies a log, and it is the one a queue cannot provide.
• Replay required — reprocessing history after a bug, or building a new consumer from the beginning.
• Throughput beyond what a database-backed queue sustains, measured rather than assumed.
• Strict ordering guarantees per key at volume.
• It is already the organisation's standard, and the marginal cost is a topic rather than a cluster. This is a completely legitimate reason and is frequently the real one.

The reasoning to generalise

Choose the simplest thing that satisfies the actual requirements, and write down the requirement that would change the answer. The failure here is not choosing Kafka — it is choosing it without being able to name which of its properties is needed.

"What does this give us that a table would not?" is the question that resolves this class of decision, and an answer that describes a future rather than a present is a decision to defer.