advanced 3 min answer

Shopify's Packwerk gem enforces module boundaries inside a Rails monolith - each package declares the packages it may depend on, a check run in CI reports references crossing an undeclared boundary, and a package_todo.yml file records the violations that already exist so only new ones are reported. Shopify's own documentation notes that Packwerk analyses static constant references and ignores method calls and objects passed at runtime. A 40-domain data platform wants the equivalent for data products. What transfers, and where does the analogy break?

shopifypackwerkpolicy as codedata meshdependencies
Show the full answer Hide the answer

The situation that produces this design

A monolith with many teams has the same problem as a 40-domain data platform: everything is technically reachable, so every dependency is implicit and nobody can change their own internals safely. Packwerk's answer is not to split the system up. It makes the dependency explicit, puts the check in the producer's own pipeline, and - critically - gives teams a way to adopt the rule without first fixing the past.

What transfers

  • Declared dependencies per unit. A data product declares the products it reads. An undeclared read becomes a reportable violation rather than a surprise during an incident.
  • The check runs in the producer's pipeline. This is the whole difference between a contract and a document, and it is the same reasoning as Packwerk's recommendation to run the check in CI rather than as an occasional report.
  • The todo file is the part people underrate. A platform with four years of history has thousands of existing cross-boundary reads. A rule that fails on all of them on day one is switched off within a week. Recording the existing set and reporting only new violations is what makes the rule adoptable at all, and the burn-down becomes a number you can track per domain.

Where the analogy breaks

Packwerk's documented blind spot is the one that matters most when you port it. It sees static references and not dynamic ones. In a data platform, dynamic is the normal case: SQL assembled from strings, templated models, notebook cells, a BI tool generating queries from a drag-and-drop model, an analyst pasting a table name into an ad-hoc console. A static parse of the transformation repository sees the governed path and misses everything else.

So the data equivalent needs two sources of truth for dependencies, not one: the declared graph from the repository, and the observed graph from the warehouse query log over a trailing 90 days. The governance signal is their difference - reads that happen and were never declared. That signal has no equivalent in the software case because the compiler already resolves almost every reference for you.

Second break: a Rails package has one owner and a data product has consumers you have never met. Blocking an undeclared read in software annoys a colleague in your own repository. The data equivalent, if enforced as a hard block, breaks an executive's dashboard at 07:00. Choose report-only mode unless a declared alternative already exists for every path you would block.

What it costs

Query-log parsing is real engineering - table and column extraction across dialects, views resolved to their bases, service accounts attributed back to humans. Budget a quarter. The todo file also has a failure mode: if nobody burns it down it becomes a permanent exception list, and the platform now documents the mess rather than reducing it. Put a per-domain target and a date on it or do not build it.

When copying it would be the wrong move

Below roughly ten datasets with real external consumers, the declared graph is a diagram on a wall and the enforcement machinery costs more than the confusion it prevents. Policy as code earns its place when the number of producer-consumer pairs exceeds what one person can hold in their head, which is somewhere between 30 and 50 pairs, not at 40 domains by itself. And if your domains cannot yet publish a dataset with an owner and a schema, boundary enforcement is premature: you are enforcing boundaries between things that are not yet products.