A data platform organises tables into bronze, silver and gold layers. What determines what belongs in each, and what is the common failure?
Show the full answer Hide the answer
What determines each layer
- Bronze: raw, as received, immutable, with provenance. No transformation beyond what is needed to land it. Its value is that it is reproducible — any downstream error can be corrected by reprocessing from bronze without re-fetching from the source, which may no longer have the data.
- Silver: cleaned, conformed, deduplicated, typed, with a defined schema. This is where quality rules and business keys are applied, and it is the layer most consumers should read from.
- Gold: aggregated and shaped for specific consumption — a dashboard, a metric, a machine-learning feature, a reverse-ETL destination.
The common failure
Layers as a naming convention rather than as a contract. Tables are placed in a layer by whoever created them, with no agreed meaning, so a "silver" table may be raw and a "gold" table may be a staging step.
The distinguishing property is the guarantee, not the position: silver guarantees a schema and a quality level; gold guarantees a definition. If a consumer cannot rely on that, the layers convey nothing.
The second common failure
Consumers reading from bronze because silver is missing something they need. That is a product signal — the silver layer does not serve them — and the response should be to extend silver rather than to accept the bypass, because a consumer on bronze is coupled to the source's raw schema and will break when the source changes.
What must be true regardless of layer count
- Reproducibility from the rawest layer, which is bronze's whole purpose.
- Lineage, so a wrong number in a dashboard can be traced to its inputs.
- Ownership per table, since a dataset with no owner is one nobody fixes.
- Compaction as a platform responsibility, because small-file proliferation from streaming writers slows every reader and no individual writer is responsible for it.
The honest note on layer count
Three is a convention, not a law. Some platforms need two, some need four, and the number matters far less than whether each layer has a stated guarantee that consumers can rely on.