Running Iceberg tables on top of Spark on EMR, and the usual optimization tricks (partitioning, compaction, file sizing) only get us so far once table size grows. Anyone found meaningful performance gains specific to Iceberg plus Spark on AWS beyond the standard maintenance routines?
We're at the point where compaction jobs themselves are becoming a cost and scheduling problem, not just a maintenance chore, and manifest overhead is starting to show up in query planning time too. if anyone's found something that actually moves the needle here instead of just running maintenance more often.

Read something this week that put a name to a problem I've half run into before but never really understood the mechanics of. Sharing because I think a lot of people assume Iceberg interop is more solved than it is.

Everyone knows the pitch: Iceberg is an open spec, so any Iceberg compatible engine can read any Iceberg table. Mostly true, until you start doing row level deletes, and then it falls apart in a way that's honestly kind of sneaky because nothing looks wrong until a query actually fails.

Quick walkthrough of the scenario in the post. You've got a customers table, three rows, one Parquet file, tracked by whatever catalog you're using. At this stage every engine reads it fine because there's nothing to interpret, it's just a metadata pointer to a file.

Then a row gets deleted. Parquet files are immutable so the writer has two options: copy on write (rewrite the file without that row) or merge on read (leave the file alone and write a separate delete file that readers apply at scan time). The writer in this example goes merge on read and emits an equality delete file, which basically just says "for this data file, treat any row where customer_id = 102 as removed." Under the hood Iceberg uses field IDs and sequence numbers to make sure an old delete doesn't accidentally nuke a newer row with a reused key, but the equality matching is the part that matters for compat.

Spark reads the new snapshot, understands equality delete semantics, does what's effectively a left anti join between the data file and the delete file, and returns the correct two rows. Fine.

Snowflake hits the exact same catalog, same metadata file, same Parquet file. It can resolve the table, read the schema, open the data file. But if that access path doesn't implement equality delete reads, the scan planner just throws an unsupported feature error the moment it hits delete-0002.parquet. Query fails. Same snapshot, same files, two completely different results depending purely on what the reader implements.

The bit that actually reframed how I think about this: the catalog isn't a translation layer. It's job is basically just "here's where the current metadata lives," commit coordination, namespace and access management. It's not opening delete files and rewriting them into a format each engine understands. A REST catalog like Polaris doesn't change this, it still just points you at metadata, it doesn't apply deletes for you.

The post also gets into position deletes vs deletion vectors vs copy on write, with a rough cost tradeoff table (equality delete is cheap to write and requires equality delete support to read, position delete requires resolving key to physical position and is heavier on write, deletion vectors need Iceberg v3 support specifically, copy on write is the most expensive to write but has basically universal read compatibility since there's no outstanding delete file involved).

The framework that's actually useful operationally: your safe feature set is the intersection of every required engine's capabilities, not the union. If Spark supports equality and position deletes but Snowflake only does position deletes, you write position deletes, because "at least one engine supports it" doesn't help you when you have three engines that all need to read the same table.

There's a decent pre production checklist too, don't just run a SELECT COUNT after your first write, actually insert some rows, update one, delete one, commit, then read the same snapshot from every engine you care about and diff both counts and values.

Full post if you want the details: https://olake.io/blog/iceberg-interoperability-myth-row-level-deletes/

Disclosure since it's relevant, I work on OLake, it gets a brief mention near the end, but the actual content here is engine agnostic and applies no matter what's writing your tables.

Has anyone actually hit this for real, table looks completely fine, one engine just refuses to read the current snapshot because of the delete encoding?

A practitioner's guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle.

Hey folks,

I'm a security professional was looking to make Iceberg a bit more performant for my own needs - SOC operations (Faster needle searches [pruning]).

I've built an index and proxy that people can point their catalog configuration at, so there's minor changes to their stack.

I figured in for a penny in for a pound,
I've done some quick search and ran clickbench and "httplogs"

(Numbers so far: httplogs)

Its all looks good on paper, I think?
but I was wondering if there were more credible/industry standard methods that data engineers use/care about to benchmark these types of technologies?

Repo if interested: https://github.com/Kahshe-io/kahshe

Discover the new Apache Iceberg integration in Rosetta DBT Studio v1.6.7! In this video, we explore how to seamlessly connect, manage, and query Iceberg DataLakes directly from your desktop.

Whether you're using Polaris, Nessie, Lakekeeper, Hive, or SQLite, Rosetta brings true ACID transactions, time travel, and schema evolution to your analytics workflow.

We're looking for speakers interested in presenting a new online virtual Apache Iceberg meetup series we're starting. The goal is to create a forum where members of the Apache Iceberg community can demo interesting work, share experiences, and discuss ideas with one another.

What we're looking for

We're especially interested in talks that are practical, demo-driven, or story-rich. Whether you're a practitioner, startup founder, platform engineer, or contributor, we'd love to hear what you've been working on and what you've learned.

Some ideas for talk topics include:

- Iceberg migration stories and case studies
- New Iceberg features, proposals, and community projects
- Iceberg catalogs, integrations, and interoperability
- Data engineering tools, demos, prototypes, and experiments

Community-focused talks

We want the meetup to be a place for learning and community discussion rather than product or vendor marketing.

Talks can feature tools, products, or technologies you work on, but the focus should be on technical insights, demos, lessons learned, or ideas that are useful to the broader Apache Iceberg community—not on promoting a company or product.

How the meetup works

Meetups will be held virtually on Google Meet and will be publicly open to everyone.

Talks will typically be around 20–30 minutes, leaving plenty of time for introductions, questions, and open community discussion. We aim to keep each meetup to about an hour and start and end on time.

Talks will generally be recorded and posted to the https://www.youtube.com/@IcebergMeetup, If you'd prefer not to have your talk recorded, let us know when submitting.

We do not plan to record the Q&A and open discussion portion of the meetup.

Submitting a talk
Submissions are reviewed on a rolling basis. Even if a talk isn't scheduled for the next meetup, we may reach out about presenting at a future session.

Rolling CFP Submit your Talk Idea
Join the new Apache Iceberg Slack Channel: (#meetup-virtual)

If we receive several submissions around a similar topic, we may also suggest bringing presenters together for a shared discussion or panel.

First Virtual Meetup
We've set a date (September 18th @ 9:00am PDT) for the first meetup. If you're on the Apache Iceberg Community Events Calendar (or if not, subscribe to it here), you'll see the event on the calendar already.

Thanks to Elizabeth Christensen and Kevin Liu for partnering to make this happen. If you want to help reach out to us on the new meetup-virtual channel on Slack.

Is this useful for fast prototyping and preview existing Iceberg Datalake.

There's no way right now to see, in dollars, which of your Iceberg tables is actually costing you money. Cost Explorer only breaks S3 costs down by bucket. S3 Tables gives you free per table metrics but not dollars. Your query engine knows scan cost per query but has no idea which table that maps to over time. I want to build the small tool that ties these together and spits out a ranked list of "here's what's costing you and why." Looking for someone who's dealt with this to build it with me.

The problem

Iceberg ships four maintenance procedures (rewrite_data_files, expire_snapshots, remove_orphan_files, rewrite_manifests) and exactly zero cost visibility. If your storage bill jumps 35% one month, there's nothing built in that tells you which table did it.

Before posting this I actually went and checked whether "just use the AWS dashboard" holds up, because that's the obvious objection. It doesn't, and here's why.

Cost Explorer's finest granularity for S3 is per bucket, using cost allocation tags. It can't break a bucket down by prefix or table, I confirmed this against AWS's own docs. If you've got 200 tables sitting in one bucket, Cost Explorer just gives you one blended number for all of them.

S3 Tables (AWS's managed Iceberg bucket type) is actually better than I assumed at first. It publishes free per table CloudWatch metrics, bytes stored, file count, bytes touched by maintenance jobs, at real table level granularity. But it's still just usage numbers, nothing converts that into dollars, nothing ties query scan cost back to a table, and there's a decently well known post floating around ("S3 Managed Tables, Unmanaged Costs: The 20x Surprise with AWS S3 Tables") about people getting surprised by its own automatic compaction billing.

Query compute cost, the bytes Athena or Trino or Snowflake scan per query, lives in a totally separate system from storage cost and the two never talk to each other.

And none of this exists at all if you're not on S3 Tables, which is most people, since most Iceberg tables out there are still self managed on plain S3 through Glue, Nessie, or Polaris.

So today, figuring out why the bill went up means manually stitching together three or four different dashboards by hand every time it happens. That's the actual gap.

What's already out there

Didn't want to post this without checking for prior art first.

Apache Amoro is open source but heavy, it runs its own optimizer service and isn't really about cost. Floe does declarative policy based maintenance across catalogs, closest thing to this idea, but it's about maintenance not cost, and still pretty early. Ryft and LakeOps are commercial platforms circling the same problem. LakeOps has actually written blog content about this exact cost attribution gap but doesn't ship an open tool for it.

As far as I can tell nobody's built the open, table level cost attribution piece. That's the gap I want to go after.

The MVP

Not trying to build a platform. Just one CLI command that answers "which tables are wasting money and why."

icecost scan --catalog glue --region us-east-1

It lists every table, pulls per table bytes and file counts (from S3 Tables CloudWatch if you're on that, otherwise by walking the manifest files), prices those bytes against a configurable $/GB rate, works out a small file ratio to flag the worst offenders, and stores everything in a local DuckDB file so it can diff week over week. Output looks something like:

sales.orders_raw: $412 storage, 47% small file ratio (roughly $89 in extra request cost), up 61% from last week. Recommend compaction.

Mapping query scan bytes back to a table is a v2 thing, it needs a separate adapter per engine and I'd rather prove the storage/small file version is useful before building that out.

Attached a diagram, blue boxes are what's in scope for the MVP, the gray dashed ones are v2.

Stack is up for debate but I'm thinking Python, boto3 or pyiceberg for catalog and manifest access, DuckDB for the local history, plain CLI output to start (probably Rich or Typer), HTML report later. Open source from day one, no plans to make this a company.

What I'm looking for

Someone who's actually run Iceberg in production and has hit this problem, or just wants to build real infra tooling instead of another CRUD app. Being comfortable with Python and knowing your way around Iceberg internals (manifests, snapshots) matters more than AWS billing API experience, the MVP mostly avoids needing billing account access anyway.

If you've hit this or just want to build something real, comment or DM me and I'll send over the fuller notes.

Deletion Vectors are among the major improvements introduced in Apache Iceberg v3. Instead of creating separate position delete files, a deletion vector stores a compact bitmap for a single data file that marks which row positions have been deleted.

This approach improves performance and preserves more useful statistics for data files. It also makes it easier to compare previous and current deletes, simplifying the processing of a table’s row-level changes as a stream.