What does a layered bronze-silver-gold structure actually buy, and where does it become ceremony?
Show the full answer Hide the answer
What it buys
1. Raw data retained, so transformation bugs are recoverable. The bronze layer keeps what arrived, unmodified. When a transformation is found to be wrong — and it will be — history can be reprocessed. Without it, the error is permanent.
2. A clear place for quality enforcement. The bronze-to-silver step is where types are enforced, ranges checked and referential integrity validated, with a defined behaviour on failure.
3. Separation of concerns. Silver is cleaned and conformed; gold is shaped for consumption. Changing a business definition touches gold; changing a source's format touches bronze-to-silver.
4. Reprocessing without re-fetching. Partners may not offer a replay, so the raw landing is frequently the only copy.
Where it becomes ceremony
When every dataset passes through all three layers regardless of need.
- A dataset with one consumer and no cleaning required does not need three physical copies.
- A dataset that is already clean and conformed at source gains nothing from a silver layer that copies it.
- Small reference data does not justify a pipeline per layer.
Each unnecessary layer is storage, compute, latency and a pipeline to operate — and the copies drift, which creates the reconciliation problem the structure was meant to avoid.
The principle that keeps it useful
Layers are a logical separation, not a mandatory physical one. The valuable properties are raw retained and quality enforced at a defined point. Whether silver is a materialised table or a view depends on whether anything reads it and whether the transformation is expensive.
The failure to design against
Quality checks that halt everything. A per-source pipeline with a broken feed should hold that source at its last known-good state and alert — not block every other source's updates, and not publish partial data either.
Scope failure per source, which is the same isolation principle that governs any multi-partner ingestion.