advanced 3 min answer

A team proposes Data Vault for a customer domain. The source is one table of 40 columns, of which 6 are business or foreign keys and 34 are descriptive attributes changing at three clearly different rates. Before agreeing, estimate how many physical tables the vault produces, how many joins a plain "current customer with address and account status" query needs, and how many rows five years of history holds for 8 million customers. Which assumption dominates the error?

data vaultestimationsatellitesjoin depthstorage
Show the full answer Hide the answer

The assumptions, stated

Three business keys are visible in those 6 key columns: customer, address and account. Attributes are split into satellites by rate of change, which is the usual guidance, and each business key gets its own satellites. Assume each customer has one address and one account, and that the 34 attributes distribute roughly 18 to customer, 10 to account and 6 to address.

The arithmetic

  • Hubs: one per business key, so 3.
  • Links: customer-to-address and customer-to-account, so 2.
  • Satellites: customer splits into three rate groups (3), account into two (2), address into one (1), and each link carries an effectivity satellite (2). That is 8.
  • Total: roughly 13 tables from one source table, a table-count multiplier of about 13 before any second source arrives. A second source system for the same customer adds its own satellites rather than merging, so 13 becomes roughly 18.

The query. "Current customer with address and account status" touches hub_customer, the two customer satellites holding the requested attributes, link_address, hub_address, its satellite, link_account and the account status satellite: 8 tables, 7 joins, and every satellite needs a latest-row-per-key predicate, which is a window function or a correlated maximum on load date. The equivalent star schema is one dimension table and zero joins.

The rows. Satellites are insert-only, so a row is written whenever any attribute in that satellite changes. If a customer's slow group changes once a year, the medium group four times and the fast group monthly, one customer produces roughly 17 satellite rows a year, or on the order of 680 million satellite rows over five years for 8 million customers against 8 million rows in a Type 1 dimension.

Which assumption dominates the error

The satellite split, by a wide margin. A satellite is rewritten as a whole row when any of its columns changes, so grouping the 34 attributes badly is what moves the estimate. Put all 34 in one satellite and every customer writes a row every time the fastest attribute moves: roughly 12 rows a year becomes 12 rows a year for all 34 columns, and the five-year figure lands nearer 1.5 billion rows, more than double. Nothing else in this estimate moves the answer that far, because the split is the only assumption that multiplies rather than adds, and it is decided by a modeller's judgement rather than measured.

The characteristic failure follows from the same arithmetic. A satellite grouped by source table rather than by rate of change looks tidy in the model diagram and degrades quietly: load times grow, the star-schema build above it slows, and nobody connects either to a decision taken in a modelling workshop eighteen months earlier. Data Vault has been in use since Dan Linstedt published the approach in 2000, and this is the failure its practitioners write about most.

What the number rules in or out

It rules out serving the vault to analysts. Seven joins and a latest-row predicate is not a query a BI tool will generate correctly, which is why every working Data Vault has a star-schema layer on top and therefore two models to keep in step. Budget the second model from day one, or the vault becomes the thing analysts route around with spreadsheet extracts.

It rules the approach in when auditability is a real obligation: when someone must be able to answer "what did we believe about this customer on 14 March, and which source told us so", and answer it from tables rather than from backups.

When this is the wrong answer

Choose the vault anyway, and ignore the multiplier, when sources are genuinely multiplying: five systems describing the same customer with more arriving from acquisitions. In that world the 13-table cost is paying for something specific, which is that new sources land as new satellites without touching existing structures or reloading anything, while a star schema pays a schema negotiation per source instead. Prefer the star schema unless both conditions hold - more than two sources describing the same entity, and an auditability requirement someone outside the data team will enforce. For a single stable source the multiplier buys nothing you can point at.