Apache Iceberg: data tables you can trust and change
Apache Iceberg is an open table format that fixes the weak spots of a classic data lake: half-finished loads, risky structure changes and no way to look back.
Many companies keep their data in a data lake, and many of those lakes are harder to run than they should be. Reports read half-finished loads, a small change to the structure takes weeks, and nobody can say what a table looked like last Tuesday. Apache Iceberg is an open table format built to fix exactly these problems.
We recently gave a lunch and learn on Iceberg for colleagues at cimt. This post is the version for readers who are not engineers: what it is, what it changes for your business, and when it is worth a conversation.
Where classic data lakes hurt
Most data lakes still organise a table like a folder on a shared drive. The way the data is split up, by day for example, is written into the folder names, and people have to keep that in order by hand.
Three problems follow. A load that fails halfway can leave a report reading half the data. A small slip in how the data is split puts it in the wrong place, and nothing warns you. And changing the structure later means rewriting the data and updating every report that depends on it, so most teams put it off. The Iceberg documentation describes these problems with folder-based partitioning in more detail.
The idea behind Iceberg
Iceberg stops treating a table as a folder. Instead it keeps a careful record of what belongs to the table: which files, which version, which structure. Think of the index in a library. Readers consult the index, not the shelves.
Every change to the table is written as a new version and published in a single step. A report sees the table either before the change or after it, never in between.
Old versions stay available until you decide to clean them up. That one design choice is behind most of the benefits below.
What it means in practice
- Loads you can rely on. A load either completes or does not appear at all, so reports never read half a delivery.
- A look back in time. You can query a table as it was last week or last quarter. That helps with audits, with reproducing a number from an old report, and with undoing a bad load.
- Changes without rework. Adding or renaming a column no longer means rewriting the data, and older data stays readable.
- A layout that grows with you. Splitting data by month can become splitting by day when volumes rise. Old data stays as it is, new data follows the new layout, and existing reports keep working unchanged.
- Less manual upkeep. Reports filter on the real date and Iceberg works out which data to read. The extra helper columns that someone has to fill in correctly are gone.
Safe changes across several tables
Project Nessie adds Git-style branches on top of Iceberg. Your team loads several tables on a separate branch, checks the results, and publishes them together only when everything looks right. Until then, nobody else sees the changes.
This matters when tables depend on each other, for example an orders table and the revenue table built from it. Nessie is optional. It earns its place when you need that cross-table safety, or separate environments for testing and production.
Is this relevant for you?
Iceberg is worth a closer look if some of this sounds familiar:
- Two reports show different numbers right after a data load.
- Changing how a table is structured is something your team avoids.
- An auditor or a colleague asks what a figure was on a given date, and nobody can reproduce it.
- Several teams change the same tables and step on each other.
If two or more of these ring true, get in touch with us at cimt. Our Data & AI team can look at your current setup with you and tell you whether Iceberg would solve a real problem or just add a project.
Technical details
For the technical details, the Apache Iceberg and Project Nessie documentation is a good place to start:
- Partitioning in Iceberg: why folder-based partitioning causes trouble and how Iceberg handles it.
- Table evolution: changing columns and layout without rewriting data.
- Queries and time travel: reading a table as it was at an earlier moment.
- Nessie and Iceberg: how the two work together.
- Nessie SQL reference: creating branches and merging them.
About the author
Further reading
Ready to apply this?
Book a conversation with cimt and see how these insights fit your data foundation.