A team runs 40 SQL scripts nightly, numbered 01 through 40 and executed in order by a shell script. Someone proposes adopting a transformation framework. A senior analyst pushes back - the scripts already run in the right order, so what does a framework actually add? What is the honest answer, and when is the analyst right?

The mechanism

Numbering encodes the order in a filename, and a filename is a second source of truth. The real dependency lives in the SQL, where script 27 reads a table script 14 created. Nothing keeps those two facts in step. A framework derives the order from the code itself: each model refers to other models by name, so the graph is a consequence of the query rather than a claim made alongside it.

That single change produces the rest:

• Correct order by construction. Adding a script is not a renumbering exercise, and the class of failure where script 31 reads yesterday's version of a table written by script 33 cannot occur.
• Parallelism for free. The graph says which of the 40 can run at once. A shell script runs them in single file, which for 40 scripts is often the difference between a 90-minute and a 25-minute build.
• Run a subtree. "Rebuild this model and everything downstream of it" is a graph operation. With numbered scripts it is a human reading SQL and guessing, which is how a partial rerun leaves two tables disagreeing.
• Environment substitution. The same code writes to a development schema for a pull request and to production at night, because the target is resolved at run time rather than hard-coded. Without it, testing a change means editing table names by hand, which is why nobody tests.
• Tests as part of the graph. A uniqueness or not-null assertion that runs in dependency order and stops the downstream build is different in kind from a query someone runs afterwards.

What it costs

A dependency, a new vocabulary for the team, and a migration of 40 scripts that will surface two or three genuine circular references the numbering had been hiding. Budget a week, not a day, and expect the build to fail loudly at first because the framework enforces things the shell script tolerated.

When the analyst is right

When the 40 scripts are stable and nobody is adding to them. If the pipeline changes twice a year, runs in 20 minutes, and one person owns all of it, a framework adds a dependency and a learning curve for benefits that only compound with change. The honest signals that the analyst has stopped being right:

• Anyone has renumbered a script in the last quarter.
• A full rebuild is the only way anyone dares to make a change.
• Two people are editing scripts and neither can say what the other's change affects.
• The build has grown past roughly an hour and single-file execution is the reason.

Common weak answers

• "It gives us version control and code review." Those come from putting SQL in git, which the team already can do with numbered scripts. Claiming them for the framework is how proposals lose credibility with exactly the person asking.
• "It is the industry standard." True since roughly 2018 and not an argument. The argument is the derived graph and what it makes possible.
• "It will make queries faster." It does not change a single query plan. Builds get faster through parallelism and incremental materialisation, which is a different claim.