A business sponsor asks for a real-time data platform because "the competition has one". Reporting is currently a nightly batch that lands at 06:00 and nobody has complained. How do you handle this?
Show the full answer Hide the answer
Do not answer the technology question
"Real-time platform" is a solution, and it has arrived without a problem attached. Answering it directly leads either to an expensive build with no measurable benefit or to a refusal that reads as obstruction. Both are avoidable.
Ask what decision changes
For each candidate use case, two questions: what decision does this data support, and what would be done differently if it were fresher?
The answers sort quickly:
| Use case | Decision | Freshness that changes it |
|---|---|---|
| Fraud screening | Decline this transaction | Milliseconds — genuinely streaming |
| Inventory shown to shopper | Whether they buy | Minutes |
| Stock replenishment | Today's purchase order | Hourly is ample |
| Executive weekly review | Steering | Nightly is ample |
| Regulatory return | Nothing | Monthly |
Usually one or two rows justify streaming and the rest do not. That is a far more useful artifact than a yes or a no, because it makes the scope small and defensible rather than making the whole platform contested.
The most common real requirement
"Real-time dashboards" very often means current when opened, not sub-second. A micro-batch on a five-minute cadence serves that at a fraction of the cost and with none of the operational burden. It is worth testing this explicitly, because it resolves a large share of these requests.
Be honest about the cost
Streaming's expense is not licences. It is always-on compute, stateful operators that must be reasoned about, harder correctness (watermarks, late data, exactly-once), harder reprocessing, and a much smaller pool of engineers who can operate it at three in the morning. None of that appears in a vendor comparison, which is why streaming platforms are frequently approved on a business case that omits their main cost.
What to propose
- Build the one case that justifies it — fraud screening — as a genuine streaming path, and let it prove the capability.
- Move the batch to micro-batch where freshness has any value. Cheap, low risk, and it addresses most of the perceived gap.
- Design so the choice is reversible. Land events into a log regardless. Batch consumers read from it in windows, streaming consumers read continuously. Adding a streaming use case later then does not require a new platform.
If overruled
Accept it and shape it. Ask for the first use case to be one with a measurable outcome, a defined success metric, and a review at six months that includes the operational cost. A streaming platform delivered with one real use case and honest running costs is a reasonable outcome; one delivered as a platform in search of users is the failure mode to avoid.