A fact table holds 3 years of data at about 40 GB per month. A nightly job rebuilds it from source. Roughly what does that cost per month in bytes processed against an incremental load, and what does the number rule out?

The assumptions, stated

Three years at 40 GB per month. The rebuild reads the full source history and writes the full table. The incremental alternative loads the new day and restates a trailing window to catch late-arriving source rows. Thirty nights in the month.

The arithmetic

Full reload:

history            = 36 months × 40 GB      ≈ 1.44 TB
per night          = 1.44 TB read + 1.44 TB written
per month          = 30 × ~2.9 TB            ≈ 86 TB processed

Incremental with a 7-day restatement window:

new data per night = 40 GB / 30             ≈ 1.3 GB
restated window    = 7 × 1.3 GB             ≈ 9 GB
per night          ≈ 10 GB read + 10 GB written
per month          ≈ 0.6 TB processed

Roughly 140 times less. At low single-digit dollars per terabyte processed - the 2026 list range for the major cloud warehouses - that is the difference between a few hundred dollars a month and a few dollars, and the gap widens linearly with every month of history added.

Which assumption dominates the error

Not the daily volume - the restatement window. Going from 7 days to 90 days takes the incremental job from 10 GB to 120 GB a night, a twelvefold increase, and it is the one parameter teams set by guesswork. Measure how late source rows actually arrive and size the window to the 99^th percentile, then handle the rare older corrections as an explicit backfill.

The second-largest term is partition pruning. If the destination is not partitioned on the column the incremental load filters by, the "incremental" job reads the whole table anyway and the saving evaporates. The arithmetic above assumes partitions are aligned with the load pattern.

What the number rules in and out

• Under about 100 GB of total history, full reload is the right answer and the cost argument is noise. Keep the simple, idempotent job.
• At 1.4 TB it is indefensible as a default, because the alternative is not complicated.
• The middle path is partition-level reload, which keeps almost all of the simplicity: recompute only the partitions whose source rows changed, by comparing a source watermark per partition. Each partition rebuild is still a full, idempotent recomputation - the property that makes full reload attractive - and the volume is proportional to change rather than to history.

The honest argument for the expensive option

Full reload is idempotent, has no late-data logic, cannot drift from source, and needs no merge semantics. Those are genuine engineering virtues and they are why the pattern persists. The answer is not to abandon them but to get them at partition granularity, which costs one watermark column and a loop.

When this is the wrong answer

When the source cannot tell you what changed. Against a system with no reliable modification timestamp and no change feed, a full comparison is the only correct way to detect deletes, and the cost is the price of correctness. The fix then lives upstream - change data capture on the source - not in the warehouse job.