intermediate 2 min answer

A team adopts bronze/silver/gold layering and every layer becomes a copy of the previous one with minor changes. What should each layer actually guarantee?

medallionlayeringdata-qualitycontractslineage
Show the full answer Hide the answer

Why it degenerates

The layer names describe a progression and do not define what changes between them. Without explicit contracts, teams add a layer whenever a transformation is needed, and the estate acquires five or six tiers of nearly-identical tables with storage multiplied, latency accumulated, and nobody able to say which one to query.

The layers must be defined by the guarantee they provide, not by their position in a pipeline.

What each layer should guarantee

Bronze — raw, immutable, faithful to the source.

  • Exactly what arrived, including malformed records, duplicates and rejected rows.
  • No business logic and no cleaning, because the guarantee is reproducibility: everything downstream can be rebuilt from bronze, and a transformation bug is fixed by reprocessing rather than by re-ingesting.
  • Append-only, with ingestion metadata: source, arrival time, batch identifier.
  • Not queried by analysts. If it is, the layer boundary has already failed.

Silver — validated, deduplicated, conformed.

  • Schema enforced, types correct, duplicates removed, referential integrity checked.
  • Conformed keys across sources, which is the substantive work: customer identifiers reconciled, code lists standardised, time zones normalised.
  • Quality rules applied with rejected records routed somewhere visible, not silently dropped.
  • The guarantee: this data is correct and consistent, and it is not yet shaped for any particular question.
  • This is the layer most analytical work should build on.

Gold — modelled for consumption.

  • Aggregates, dimensional models, feature tables, metrics — shaped for a specific audience and a specific set of questions.
  • The guarantee: this answers these questions correctly and efficiently, and it is expected that several gold tables exist over the same silver data.

The rules that keep it from degenerating

  • A new layer requires a new guarantee. A transformation that does not change what is guaranteed belongs inside an existing layer, not in a new one.
  • No layer skipping — gold reads silver, not bronze — or the guarantees stop meaning anything and lineage becomes unreadable.
  • No business logic in bronze, ever, since that is what makes reprocessing possible.
  • Silver is the contract boundary between the data platform and its consumers, and should be treated as a published interface with schema stability.
  • Materialise where it pays. Not every layer needs to be a physical table; a view is a legitimate implementation of a layer where the transformation is cheap, and it avoids the storage and latency cost.
  • Ownership per layer per domain, so someone is accountable for the guarantee.

The test: for any table, can someone state what it guarantees and who owns it? Where the answer is "it is silver, so presumably it is clean", the layering is decorative.