Model Graph Pruning

Transformation projects grow monotonically. Models are added continuously, each addition individually justified, and removed never — because removal delivers nothing visible and requires proving that nothing depends on the output.

A meaningful proportion of scheduled analytical work produces artefacts nothing reads, and finding them is substantially cheaper than making them faster.

Why it matters

Every model in the graph costs compute on every run, lengthens the build, adds a possible failure point, and adds to the surface anyone must understand. The aggregate cost is nobody's decision, which is the same structural failure as feature-flag accumulation and unused-index accumulation.

Implementation patterns

• Usage telemetry on the outputs, not just on the models. Which tables were queried, by whom, in what period — without this the pruning decision is a negotiation rather than an inference.
• Cost and duration per model, so the expensive ones are visible.
• The two lists that follow: expensive with no consumers are the first deletions; expensive with many consumers are the first materialisations.
• A named owner for the graph as a whole, since per-model ownership does not produce anyone accountable for its total size.
• Periodic review, quarterly, treated as routine maintenance rather than as a project.
• A deprecation path for models with consumers, since removing one that a dashboard silently depends on is how the practice loses trust.

The adjacent optimisations the same data enables

• Incremental materialisation for the large models, with a defined strategy for late-arriving and updated records — the part that makes incremental hard and that is frequently handled by hoping.
• Materialising shared expensive intermediates rather than recomputing per consumer.
• Flattening depth, since a wide graph parallelises well and a deep one does not.
• Tests concentrated on the models that matter, since a build running a thousand assertions on unimportant tables spends its time in the wrong place.

Industry example

Analytics organisations using transformation frameworks such as dbt reliably reach hundreds of models within two years, at which point the build's duration becomes a delivery constraint. The intervention with the best return is almost always deletion rather than optimisation, and it is available in a day once output usage is instrumented.

Failure scenarios

• No output usage telemetry, making pruning a debate.
• No owner for the graph, so its size is nobody's problem.
• Deleting a model a dashboard silently depended on, which ends the practice.
• Optimising expensive models that nobody consumes.
• Adding models with no removal process, which is the default.

Trade-offs

Pruning risks removing something that is used rarely but genuinely — a quarterly report, an annual regulatory extract — which a short observation window will not see.

The mitigation is an observation window long enough to include the periodic consumers, plus a deprecation notice rather than an immediate deletion. That costs a quarter and it is what makes the practice safe enough to repeat.

Interview question

"Your transformation build takes four hours. Before optimising anything, what would you measure, and what would you expect to be able to delete?"