A startup with 3 data engineers and 40 GB of total data proposes this platform: raw, bronze, silver, gold and platinum layers, each a full copy in a lakehouse, with a data quality gate and a catalogue entry per layer, for 12 source tables. Review it. What would you remove, change and leave alone?
Show the full answer Hide the answer
What is actually required
Three things: the original data kept as received (so a bug in transformation can be replayed), cleaned and conformed data that analysts trust, and a few business-ready tables with stable definitions. That is two or three layers, and the third is often views rather than copies.
What I would remove
- Platinum and the separate raw layer. Raw and bronze are the same promise (unmodified source data); keeping both doubles storage and adds a handoff nobody owns. Platinum has no consumer yet, and a layer without a named consumer is cost without a contract.
- Full copies for layers that are pure projections. If gold is a filter and a join over silver, make it a view or a lightly materialised table. Each physical copy is a freshness delay and a place for drift.
- A catalogue entry and quality gate per layer per table. That is 60 gates for 12 tables. Gate where a defect becomes expensive: on entry to silver (types, keys, duplicates) and on the few gold tables that feed finance.
The one change that matters
Write down what each layer promises and who reads it. A layer is a contract: bronze is append-only and unmodified, silver is deduplicated and typed with a primary key, gold has a documented grain and owner. Without these, layers are folders. With them, three layers are enough and a fourth must justify itself by naming a consumer and a promise that silver cannot make.
What I would leave alone
Using an open table format and object storage even at 40 GB looks heavy but is cheap, avoids a migration later and gives time travel for debugging. Daily orchestration with a lightweight tool is fine. Also leave the instinct to keep the original data: replay is the cheapest insurance a data team has.
How the cost shows up
Each extra layer adds roughly one pipeline to run, one failure point, one place a late file can stall, and one more thing a new hire must learn. For three people, the dominant cost is attention, not storage: at 40 GB the storage bill rounds to nothing.
When the full design would be right
Hundreds of source tables, several teams producing and consuming, regulated lineage needs, or a platform team serving others. There, separate raw and curated zones with different access rules, and a published layer for external consumers, earn their keep.
How I would argue it in the review
Medallion is a naming convention for lakehouse layers, not a requirement, so a team may use fewer. Ask the proposer one question per layer: "who reads this, and what do they rely on that the previous layer does not give them?" If the answer is "nobody yet", defer the layer. It costs a day to add later and a week to remove once people depend on it.
Common weak answers
- "Medallion is best practice, so keep all five." It is a pattern with a cost, and the cost is paid by this team.
- "Drop layers and query the source." Losing the raw copy removes replay.