A 40-person company has 80 GB of analytical data growing about 4 GB a month, six analysts, a nightly load from Postgres and four SaaS sources, and no data engineer. An architect proposes an object-storage lakehouse with an open table format so the company never has to migrate again. What should they build first?
Show the full answer Hide the answer
The deciding property
80 GB fits in the memory of a single large machine, and there is nobody to operate a platform. Those two facts settle it. Every argument for a lakehouse is an argument about scale, engine choice or storage cost, and at 80 GB none of the three is binding: storage is a rounding error, one engine is enough, and a table scan is seconds.
Why a managed warehouse
It has no maintenance workload class. A lakehouse has one, permanently: compaction, snapshot expiry, clustering, a catalogue to run, and a query engine whose version you now own. That is a standing job of perhaps a day a month, and in a company with no data engineer it becomes the analyst who happens to be curious, then nobody, and the symptom is queries that slow by 3 to 6 times over a year with no change in data volume.
The claim that an open format means never migrating again is the wrong model of migration cost. Migrations are expensive because of accumulated SQL, report definitions, extracts and consumers, not because of file format. Keeping files portable saves the cheapest part of the eventual move.
What would flip the decision
| If this changes | Choose | Because |
|---|---|---|
| Data passes roughly 10-30 TB or per-TB storage dominates the bill | Lakehouse | Warehouse storage pricing and loading stop being free at the margin |
| Non-SQL workloads appear - training, feature engineering, image or log blobs | Lakehouse | Multiple engines need one copy of the data |
| A second engine or a second vendor must read the same tables | Open table format | The format becomes the integration contract |
| A regulator requires data to stay in storage the company controls | Lakehouse | Residency and custody beat convenience |
| Still 80 GB in two years | Stay | Nothing has changed except fashion |
Why the other options fail
- The lakehouse now. It is the right answer two orders of magnitude later. Adopted here it buys optionality nobody can exercise and costs the only scarce resource in the company - attention.
- A lake of Parquet files with a detached engine. Cheapest to start and it has no atomic commits, no schema enforcement and no cheap deletes, so the first concurrent write or GDPR erasure request becomes a project.
- Querying the production replica. Right for a one-off question against under 10 GB. Wrong as the platform: six analysts scanning a replica that also serves failover will produce replica lag that reaches the application, there is no history once rows are updated in place, and the four SaaS sources have nowhere to land.
When this is the wrong answer
When the data has nothing to do with the volume. 80 GB of regulated clinical or payment data with jurisdiction and custody constraints can justify object storage the company controls on day one. The deciding property is then legal, and the sizing argument above is irrelevant.