intermediate 2 min answer

A startup with 3 data engineers and 40 GB of total data proposes this platform: raw, bronze, silver, gold and platinum layers, each a full copy in a lakehouse, with a data quality gate and a catalogue entry per layer, for 12 source tables. Review it. What would you remove, change and leave alone?

medallionover-engineeringlayersdata qualitysmall teamreview
Show the full answer Hide the answer

What is actually required

Three things: the original data kept as received (so a bug in transformation can be replayed), cleaned and conformed data that analysts trust, and a few business-ready tables with stable definitions. That is two or three layers, and the third is often views rather than copies.

What I would remove

  • Platinum and the separate raw layer. Raw and bronze are the same promise (unmodified source data); keeping both doubles storage and adds a handoff nobody owns. Platinum has no consumer yet, and a layer without a named consumer is cost without a contract.
  • Full copies for layers that are pure projections. If gold is a filter and a join over silver, make it a view or a lightly materialised table. Each physical copy is a freshness delay and a place for drift.
  • A catalogue entry and quality gate per layer per table. That is 60 gates for 12 tables. Gate where a defect becomes expensive: on entry to silver (types, keys, duplicates) and on the few gold tables that feed finance.

The one change that matters

Write down what each layer promises and who reads it. A layer is a contract: bronze is append-only and unmodified, silver is deduplicated and typed with a primary key, gold has a documented grain and owner. Without these, layers are folders. With them, three layers are enough and a fourth must justify itself by naming a consumer and a promise that silver cannot make.

What I would leave alone

Using an open table format and object storage even at 40 GB looks heavy but is cheap, avoids a migration later and gives time travel for debugging. Daily orchestration with a lightweight tool is fine. Also leave the instinct to keep the original data: replay is the cheapest insurance a data team has.

How the cost shows up

Each extra layer adds roughly one pipeline to run, one failure point, one place a late file can stall, and one more thing a new hire must learn. For three people, the dominant cost is attention, not storage: at 40 GB the storage bill rounds to nothing.

When the full design would be right

Hundreds of source tables, several teams producing and consuming, regulated lineage needs, or a platform team serving others. There, separate raw and curated zones with different access rules, and a published layer for external consumers, earn their keep.

How I would argue it in the review

Medallion is a naming convention for lakehouse layers, not a requirement, so a team may use fewer. Ask the proposer one question per layer: "who reads this, and what do they rely on that the previous layer does not give them?" If the answer is "nobody yet", defer the layer. It costs a day to add later and a week to remove once people depend on it.

Common weak answers

  • "Medallion is best practice, so keep all five." It is a pattern with a cost, and the cost is paid by this team.
  • "Drop layers and query the source." Losing the raw copy removes replay.