Data Lakehouse

Quick answer:

A data lakehouse is a data lake that behaves like a warehouse. You keep your data in cheap open files (usually Parquet on S3 or similar), then add a table format layer that gives those files database powers: transactions, schema enforcement, updates, and fast SQL. One copy of the data serves analytics, machine learning, and everything between.

It’s the industry’s attempt to end the “lake for data science, warehouse for analytics, sync headaches between them” split that defined the 2010s.

What is a lakehouse?

Start with a data lake: files in object storage, cheap and open but chaotic. The historical problem was that files can’t do what databases do. No transactions, so 2 jobs writing at once corrupt each other. No enforcement, so a bad upload silently breaks the schema. No efficient updates, so fixing one row means rewriting whole files by hand.

The lakehouse fixes this with a metadata layer called an open table format. Apache Iceberg (built at Netflix), Delta Lake (built at Databricks, open sourced in 2019), and Apache Hudi (built at Uber) all do roughly the same job: they track which files make up a table, which version is current, and what the schema is. Suddenly your pile of Parquet behaves like a proper database table, with ACID transactions and even time travel to past versions.

On top of that sits any query engine you like. That’s the strategic point: the data lives in an open format that you own, and engines compete to query it.

Who invented the term?

Databricks pushed “lakehouse” into the mainstream with a January 2020 blog post and a research paper the following year. It was partly genuine architecture and partly marketing: Databricks had the lake-side platform and needed a word for “you don’t need to buy Snowflake next to us”.

The funny part is how completely the framing won. Snowflake now supports Iceberg tables, BigQuery reads open formats through BigLake, and every warehouse vendor quietly adopted the architecture their rival named.

How is a lakehouse different from a warehouse?

Three real differences:

  • Who owns the storage format. In a classic warehouse, data lives in the vendor’s proprietary format; leaving means exporting everything. In a lakehouse, data is open files in your own cloud bucket; leaving means pointing a different engine at the same files.
  • Workload range. Warehouses are SQL machines. Lakehouses also feed Spark jobs, ML training, and streaming from the same tables.
  • Operational effort. This is the honest downside. A warehouse hides file compaction, clustering, and cleanup behind the product. In a lakehouse, someone has to run that maintenance or query speed decays.

When should you pick a lakehouse?

If you have a real machine learning practice, petabyte-scale raw data, or a hard requirement to avoid vendor lock-in, the lakehouse is the sane default, and Databricks is the reference implementation.

If your workload is business analytics for a mid-size company, a plain warehouse is still less work. You’ll trade some theoretical openness for a lot of practical simplicity, and for most teams that’s a good trade. We’ve watched plenty of 20-person data teams adopt lakehouse architecture they didn’t need because the blog posts were persuasive.

What are the benefits and drawbacks of a lakehouse?

Benefits of a lakehouse

One copy of the data, every workload

SQL analytics, Spark jobs, ML training, and streaming all read the same tables. The old lake-to-warehouse sync pipelines, with their lag and drift, simply stop existing.

You own the storage format

Iceberg and Delta tables are open files in your own bucket. Switching query engines becomes a repointing exercise instead of a data migration, which changes your posture in every renewal negotiation.

Warehouse behavior at lake prices

ACID transactions, schema enforcement, and time travel over object storage that costs cents per gigabyte. For petabyte-scale estates, the storage line item alone can justify the architecture.

Time travel and audit built in

Table formats keep snapshots, so “what did this table say last Tuesday?” is a query parameter, and a bad write can be rolled back instead of restored from backups.

Drawbacks of a lakehouse

You inherited a database’s chores

Compaction, snapshot cleanup, clustering: warehouses hide this maintenance inside the product, and a lakehouse hands it to your team. Skip it and query performance rots quietly.

More moving parts, more expertise

Table format plus catalog plus query engine plus storage layout is a deeper stack than “log into Snowflake”. The talent bar is real, and small teams feel it.

The format race has losers

Betting on a table format in the years Iceberg, Delta, and Hudi were fighting meant migration risk. Iceberg’s momentum has calmed this, but interop between formats still isn’t free.

Fine-grained SQL performance still trails

For classic BI (small, chatty dashboard queries), a tuned warehouse is often still snappier than an engine reading open files. The gap narrows every year; it hasn’t closed.

Where does this all land?

The likely end state: open table formats (Iceberg looks like the winner of that race) become the neutral storage standard, and “warehouse vs lakehouse” stops being an architecture decision and becomes a pricing decision between query engines. Snowflake, Databricks, and BigQuery all reading the same Iceberg tables already mostly works today.

When the storage is neutral, the fight moves to compute, governance, and who makes the maintenance invisible. Watch that space, because it’s where your bill will be decided.

Avatar photo

Panoply

Panoply wrote for the Panoply blog.