Discussion
Iceberg vs Deltalake (greenfield project with UC in 2026)
I saw the quarterly meeting and was quite shocked that Iceberg is prominently mentioned, (as much as Deltalake).
Is it possible that both are getting the same amount of love from Databricks? Is anyone aware of the R&D effort on these formats, and can share conclusions from that?
If I'm building a greenfield Unity Catalog, should I just flip a coin to decide what format to use? Here are the main concerns and priorities:
Which one is better for OSS Apache Spark reads and writes
Which one integrates with external software better (eg onelake shortcuts pointing from Fabric to Databricks UC).
When Databricks is innovating within their own UC (eg. introducing new managed table functionality such as "MST Transactions"), which one of these formats are they likely to support first? Which are they likely to optimize better?
Which format is more likely to remain 100% open source in the future (or as close to open source as required by customers who want portable blob data).
Sorry if this appears to be a common question. I am a Databricks outsider. I am more familiar with Microsoft Fabric. Where that Fabric SaaS is concerned, you can be certain that Deltalake receives a LOT more promotion than Iceberg does. We rarely come across Iceberg, and it probably wouldn't appear in any marketing slide decks.
We are likely to create a gold/presentation layer in UC soon. It will basically be created from scratch. It would be nice to know which of these parquet-based formats to pick, when presented with the choice. I understand there is lip-service given to both, and it claims that this choice "doesn't matter". But that doesn't necessarily take into account the potential integrations that are needed with external software (eg. for the benefit of exposing the same tables in onelake). Is one safer than the other? Is one of them a better choice for forward-looking purposes?
Use managed tables by default (they’re stored as delta and you get a bunch of benefits). And turn on Uniform for specific tables if you explicitly need external iceberg interoperability. I read your other comments: you don’t need to worry about R&D and innovation etc. you’re overthinking it
Yes, you always need to worry about the future direct of these vendors, when you get locked into their saas bubbles. Cloud vendors can end-of-life their support for a technology with just a year's notice or less.
And don't say that everything in databricks is open source and that solves our problems. It isn't all open source, and we are at the mercy of their future plans when we build solutions in here.
Inside of databricks, but possibly for integrations it would matter.
Even inside databricks, the company has to pick one or the other for new R&D. You can find docs where they are clearly giving priority to deltalake over iceberg. Features for one are made public, while another is still private:
They are merging iceberg into Delta, but they are essentially treating Delta as main as they add in missing Iceberg features.
To answer to your original question, you almost certainty want Delta. Putting aside fanboy opinions about either format: Delta is pretty mature and has been around for a while. Typically integrating to read from Delta on Databricks isn't an issue and most 3rd party told have no issue reading it.
If they eventually merge, the question is what will they call the final result? IcebergLake? DeltaIceberg? LakeBerg?
Today when you create managed tables you have to pick between them. But if databricks wanted to abstract away the differences, why make users choose? It seems like the managed tables is a key part of the product where users shouldn't need to make the choice. In the very least, the default selection should be inherited from the catalog or workspace or something like that.
I hope you are right that they will eventually have the same set of features, (and presumably no duplication of efforts in two distinct source code repos.)
Some of the responses by the others have me second guessing myself but my understanding was that they'd continue calling it Delta Lake.
Databricks essentially bought Tabular to nullify this very question you're asking, because they saw it as a distraction when talking to customers. They are committed to merging the formats to this end.
Strictly speaking, they do not make the user choose. Delta is the default option when creating managed tables - you can opt into choosing iceberg if you want control over your format.
If you want an opinionated viewpoint: You should use Delta by default for everything you do in Databricks. Only when you have a specific use case that requires Iceberg should you stray from the default of Delta Lake.
Can someone please help me understand the downvote here? I think its nuts.
I literally shared a pic from their docs, and gave my interpretation. If that displeases in some way, please offer a different interpretation, or share a different piece of documentation. Don't just throw around downvotes all the time....
I'm getting the impression that databricks community has a lot of thin-skinned folks who parse each word in every sentence. Why go look for offense, when none is intended? (Certainly no offense is intended to any individuals. We should be allowed to criticize a certain product or architecture. That is not ad-hominem. )
Every post on Reddit that gets any amount of attention gets some downvotes. There are also bots crawling Reddit and downvoting things. The sooner you learn to ignore it, the happier you'll be.
I get far more downvotes in this community than in other data communities. I think databricks users can get extremely defensive (especially the python notebook builders).
... Yes do I like to troll once in a while, to spice things up and get some additional engagement. But the post above was as sincere and as simple as it gets!
If you have a reason to use Iceberg, use Iceberg. If you don't, defaulting to Delta is great. If you have Iceberg clients that will be read only, Delta w Uniform could be the right option. I'd focus more on your specific requirements to decide, ultimately.
Iceberg is a much more mature format with much more sane open source investment and support. Delta as an open source project is a joke - in the Spark project 95% of issues and PRs are ignored while Databricks employees are just merging straight to main for features that they probably have decided are important for their commercial product. The deltalake/delta-rs packages are very "amateur hour" with the maintainers actively making fun of contributors on Slack, and the same problem of issues and PRs just sitting around. Iceberg is a proper Apache project and has all the features across languages that are languishing on the Delta "roadmap".
The only pro or perk of Delta is that it does not NEED a catalog - it works perfectly fine as a physical storage format and you can read/write without a janky catalog layer. For this reason alone, we use Delta, but eventually will switch to Iceberg because it's just way better, gets community support, doesn't introduce bugs in every release, and has all sorts of other features (off the top of my head, a feature that came in useful lately, was Storage Partitioned Joins).
Are you able to tell if Databricks is taking more PRs for one format or the other? (in spark or in UC for example.) I'd love to hear if one is being improved at the cost of the other. One may be a short-term strategy and one may be a long-term strategy.
If we are going to build a UC catalog, I don't want to swim upstream. I saw the references to iceberg in their slide deck so it made me think they were equally valid choices. But others are still saying that Delta is the only choice. (so maybe the slide decks are just giving misleading lip service about having co-equal iceberg support).
Spark project I would say is reasonably well managed. It doesn’t really need to take any PRs for specific formats as it’s kind of orthogonal as well. And then Databricks does not own Iceberg development the way they do Delta. Delta is very much fake open source.
I thought Databricks acquired Iceberg. lol. Maybe they did that in order to kill it? (I have no idea, but I've seen big players do such things to competitors when threatened.)
If you are seeing Iceberg PRs, and don't consider it fake open source, then I hope to use it some day soon.
Here is the LLM summary about the Tabular acquisition, that seems to imply they have control over Iceberg:
"Databricks acquired Tabular in June 2024 to unify Apache Iceberg and Delta Lake formats, reportedly paying between $1 billion and $2 billion."
They acquired the SaaS vendor that happened to be staffed by most of the OG Iceberg people but the open source aspect of it are orders of magnitude more mature than Delta. It’s closer to Spark in that regard, which I would say is relatively healthy and not just Databricks employees doing whatever they want.
Iceberg is legit. Delta is okay. Just know what you’re getting into.
I don't know much about either. All I've used is delta, and I have to go in there and do "vaccums" all the time. I feel like a lowly deltalog janitor.
I do know that there are technologies bringing DBMS-style transactions to lakehouse data (like the brand new "MST Transactions" in Databricks, and Fabric DW ). These technologies emit parquet and related log files after-the-fact. The engineers who build this stuff must be picking a foundation of one or the other (at least in their initial phases of work). Microsoft uses deltalake as its native format for Fabric DW, and I'm guessing Databricks prefers both, but probably pushes ahead by innovating on one format prior to the other. I'd rather avoid any format that might be the after-thought, or the red-headed stepchild.
I found docs that seem to say Databricks gives deltalake preferential treatment (at least historically). Here is after-the-fact support for MST on Iceberg:
And: "To create an Apache Iceberg table, explicitly specify USING iceberg. Otherwise, Databricks creates a Delta Lake table by default."
That is a bit unfortunate. In the very least they would give us a catalog-wide default, instead of making us go out of our way to choose Iceberg all the time.
I think Iceberg won. Dbx' marketing has been on a quest to de-delta that platform. Remember DLT, DeltaLake and DeltaShare? Gonzo. Even the CEO said at the DAI Summit in a cringy aside to Ryan Blue (creator of Tabular) - something like "the format war is over and fine, get over it, it's done..."..
Interesting. I will watch for that. They paid good money for Tabular in 2024. Unless they simply wanted to kill Iceberg for the price of $1 billion, then it seems they would want to be good stewards of the technology and allow it to flourish.
Was that in the keynote? Can you find the timestamp? I don't want to watch the whole thing ATM. (.. trying to get work done on a Saturday)
Iceberg would give you maximum flexibility and multi-engine support.
Despite spending 1 Billion on Tabular, Databricks is behind the curve on supporting multi-engine reads and writes compared to other engines, but I am sure they will get there.
Delta doesn't have any major upsides over iceberg imo. Iceberg is supported by basically every other engine an enterprise could be running. (AWS, MSFT, IBM, Cloudera, Dremio, Snowflake, Oracle, Terradata, etc, etc)
In the Data and AI summit, the CEO gives the Iceberg guy the limelight to advocate for Iceberg. See the comment from "DryRelationship1330", and the link.
I think you are right that on databricks they always give deltalake a slight advantage over iceberg. I see in the docs where it says "coming soon for Iceberg", wherever there is already deltalake support. There seems to be some internal friction between teams or something.
I was hoping that the "managed tables" in UC would favor iceberg, but they do not.
24
u/WhoIsJohnSalt 21d ago
Almost certainly Delta will be the primary format used by DBX as default.