A platform pushes warehouse-derived data back into operational systems. What are the risks, and what must be designed?
Show the full answer Hide the answer
What it is for
Making analytical results actionable — a computed customer segment, a churn score, a usage summary — available in the operational systems where people and processes act on them.
The risks
1. Stale data driving operational decisions. Warehouse data is batch and lagged. A segment computed overnight and pushed into a system that acts in real time means decisions are made on yesterday's state. The freshness must be visible to the consumer, or it will be assumed current.
2. The warehouse becoming an operational dependency. If an operational system cannot function without a warehouse-derived field, the warehouse's availability requirement has silently changed from analytical to operational — with no corresponding change in how it is operated or monitored.
3. Write amplification onto operational systems, where a full sync of a large derived dataset overwhelms a system sized for transactional load.
4. Rate limits and API quotas on the destination, particularly for external systems.
5. Ownership confusion. A field in an operational record that is written by a pipeline, not by the application, is a source of confusion for every engineer who later wonders where it came from.
What must be designed
- Incremental sync, sending only changes, with idempotent upserts keyed on a stable identifier.
- Rate limiting and backoff toward the destination, respecting its quotas.
- Freshness metadata alongside the value, so consumers know how stale it is and can decide.
- A defined behaviour when the pipeline is behind — serve stale, or indicate unavailable, decided deliberately per field.
- Clear marking of pipeline-written fields in the operational schema, so their provenance is obvious.
- Monitoring of sync lag as a first-class SLI, since a stalled sync means operational decisions on increasingly old data with no error anywhere.
The architectural question worth asking first
Should this computation be operational rather than analytical? If a value is needed in real time and drives operational behaviour, computing it in the operational path — or in a streaming pipeline — is frequently better than computing it in the warehouse and pushing it back.
Reverse ETL is the right answer when the computation genuinely needs the warehouse's data and the staleness is acceptable. It is the wrong answer when it is being used to avoid building the operational capability.