Lakehouse

Definition

A data lake is files in object storage: cheap, open, and lacking transactions, schema enforcement or efficient updates. A lakehouse adds a table format layer over those files that provides atomic commits, snapshot isolation, schema evolution, time travel and row-level updates and deletes.

The problem it solves

Classic lakes had a specific painful failure: updating data meant rewriting whole partitions, which was slow, expensive, and not atomic — readers could see a partially rewritten partition. Deleting one person's records for a privacy request meant rewriting terabytes.

Uber's development of an incremental table format came directly from this pressure. Their data was arriving continuously and being corrected continuously — a trip's fare adjusts, a rating arrives late, a dispute changes a record days afterwards — and full-partition rewrites could not keep up with either the volume or the freshness requirement.

The architectural idea that matters is incremental processing: consumers can ask "what changed since the last time I looked" rather than rescanning everything. That single capability collapses the cost of keeping downstream tables fresh, and it is what makes near-real-time analytics over a lake economically possible rather than merely technically possible.

Implementation patterns

• A medallion-style progression — raw, cleaned, curated — so reprocessing from raw is always possible and each layer has an owner.
• Partitioning aligned to query filters, usually by date, with attention to file sizes.
• Compaction as a scheduled job. Streaming ingest produces many small files, and small files are the dominant performance problem in lake-based systems. Compaction is not optional maintenance; it is a core operational task.
• Time travel for reproducibility, so a model trained last month can be retrained on exactly the data it saw.
• Row-level deletes to satisfy privacy requests without partition rewrites.

Failure scenarios

• The small-file problem, unmanaged: millions of tiny files make metadata operations and query planning slower than the scan itself.
• Schema drift accepted silently, so a column's type changes upstream and downstream queries return nulls rather than errors.
• Choosing a lakehouse for a workload that is a warehouse workload — modest volume, heavy BI, strong governance needs — and taking on the operational burden of compaction, clustering and format upgrades for no benefit.
• Open format chosen for portability, then locked to one engine's extensions anyway.

Trade-offs

Bought: cheap object storage economics, open formats readable by multiple engines, one copy serving both analytics and machine learning, incremental processing, and time travel. Sold: a warehouse's maturity in governance and query optimisation, and a meaningful operational burden — compaction, clustering, metadata management and version upgrades are ongoing work that a managed warehouse hides.

Interview question

"When would you choose a warehouse over a lakehouse for a new analytics platform, and what would change your mind?"