Review this estate. Postgres and three SaaS sources land raw in bronze. A Data Vault of hubs, links and satellites is built in silver. Star schemas are built in gold. Six departmental extracts are then copied into departmental warehouses. Freshness is 40 minutes, the team is six people, and there are 42 dashboards. What would you remove, what would you change, and what would you leave alone?
Show the full answer Hide the answer
What is actually required
Forty-two dashboards over four source systems with a six-person team. The requirements that justify structure here are: traceability back to source (so a disputed number can be explained), a stable place to encode business logic once, and freshness in the tens of minutes. Nothing in that list requires five physical copies of every row.
What I would remove, and why it is safe to
- The six departmental warehouse copies. They exist because someone wanted isolation of cost or access. Both are achievable with separate compute on the same storage and with row-level policies, and each copy is a place where a number can differ from the certified one. Removing them removes the entire class of "which warehouse is right" incidents.
- The Data Vault layer, unless there is a regulatory driver nobody mentioned. Vault buys auditability and load parallelism across many source systems with unstable schemas. Four sources, six engineers and no stated audit obligation does not pay for hubs, links and satellites, and the cost is visible: analysts cannot read the model, so gold becomes the only usable layer and silver is pure overhead. Replace it with a documented staging-to-model transformation carrying source system and load timestamp on every row, which preserves the traceability that was the real requirement.
The one change that matters
Make the gold star schemas the only published interface, and define the metrics once above them. The estate's actual problem is not layers, it is that 42 dashboards can each define revenue. Consolidating layers without consolidating definitions moves the inconsistency rather than removing it.
What I would leave, even though it looks odd
- Bronze as an immutable raw landing zone. It looks redundant and it is the thing that makes reprocessing possible when a transformation turns out to be wrong. Keep it exactly as raw as it is; the instinct to clean on landing is what makes replay useless.
- 40-minute freshness. Nobody asked for real time. Resist the offer to make it streaming; the cost is permanent and the benefit is unclaimed.
How I would argue this in the review
Not as "this is over-engineered", which loses the room. As a question of what each layer guarantees to its consumer. If two layers make the same guarantee, one of them is a copy with a maintenance bill. Then price it: each layer is roughly a person of ongoing work in a six-person team, and the team's real constraint is people, not compute.
When this critique would be wrong
A bank consolidating 30 source systems with regulator-facing lineage obligations has exactly the conditions Data Vault was designed for, and there the load parallelism and the insert-only-satellite history are worth their complexity. The shape of the criticism is not "vault is bad" — it is that this estate has four sources and a six-person team.