An organisation running Kafka at trillions of messages per day concludes the coordination and rebalancing model no longer fits and builds a replacement. What evidence should exist before replacing a proven system rather than tuning it?

The default should be strongly against replacement

A proven system carries enormous accumulated value that does not appear in any comparison: years of bug fixes for conditions nobody remembers, operational knowledge, tooling, integrations, a hiring pool who know it, and a community finding problems on your behalf.

A replacement starts with none of that and must re-earn all of it while the organisation depends on it. The historical base rate for "we will build a better one" is poor, and the failures are expensive.

So the burden of proof sits firmly on replacement, and the question is what would discharge it.

The evidence that would justify it

• A limit that is architectural rather than parametric. Tuning, sharding, adding clusters and upgrading hardware have been exhausted, and the remaining constraint is in the system's design — a coordination model, a metadata structure, a rebalancing protocol whose cost grows with cluster size or partition count in a way no configuration changes.
• Quantified operational pain, sustained. Not "rebalances are annoying" but the measured cost: unavailability during rebalance, hours of engineer time, incidents attributable to the mechanism, and the frequency of all three over a long period.
• A scale trajectory that makes it worse, so the problem is growing rather than static.
• The upstream path exhausted. Has the problem been raised with the project? Are fixes in progress? For an open-source dependency, contributing the fix is dramatically cheaper than replacement, and it must be shown to be infeasible rather than merely slow.
• A clear statement of what the replacement does differently, at a design level, with an argument for why that resolves the constraint — and ideally a prototype demonstrating it under representative load.
• The organisational capability to own it forever: a team that can operate, evolve and staff a foundational distributed system indefinitely. This is usually the binding constraint and is usually underestimated.
• A migration path that is incremental and reversible, because a big-bang cutover of a foundational system is not an acceptable risk at any scale.

The organisations for which this is ever true

Very few. The argument requires that you operate far enough beyond the system's design envelope that the design itself is the limit, and that you can fund a permanent team. At most organisations the honest finding is that the system is fine and the configuration, the partitioning strategy or the client behaviour is the problem.

LinkedIn's move toward purpose-built log storage — after operating Kafka at extreme scale for many years, having originally created it — is a legitimate instance precisely because those preconditions hold: an unusual scale, a demonstrated architectural constraint, and the capability to own the result.

What to do instead, in order

1. Measure precisely what is failing, and confirm it is the system rather than how it is used.
2. Tune, re-partition, upgrade, and split into more clusters, which addresses most coordination-related pain by reducing per-cluster scale.
3. Contribute upstream or fund the work.
4. Adopt a newer mode of the same system where one exists that addresses the constraint.
5. Replace one workload, not the platform, and learn from it in production.
6. Only then consider general replacement.

And write down, before starting, what would prove the replacement was a mistake — because the sunk-cost pressure on a foundational rewrite is immense and the exit criteria are unwritable afterwards.