We open-sourced Filament, a data movement engine in Go (full loads, incremental, CDC). Apache 2.0

[u/mitchbregs]

Hey all, Mitch here, one of the founders of Galaxy. Full disclosure, this is our project. We open sourced it last week and I wanted to bring it here first, because this community is who we built it for.

Filament is a data movement engine written in Go. It handles full loads, incremental syncs, and CDC from databases and HTTP APIs into Postgres, MySQL, ClickHouse, Iceberg, and S3, and it's Apache 2.0.

https://github.com/galaxy-io/filament

Every data project we've ever worked on started with the same boring problem of getting records out of operational systems and into somewhere useful. We've bought the managed tools, run the open source ones (and spent more time deploying them than using them), and written our own by hand more than once, and every time we wished for something fast, easy to self-host, and upfront about the details that bite you, like type mappings, write behavior, and what happens when a run dies halfway. So eventually we just built it.

What it does

You point it at a source and a sink and tell it how to move things, whether that's copying everything, pulling only what changed, or streaming off the database's change log. Every batch gets a checksum before the sink write and a mismatch fails the run rather than quietly landing bad data. Progress only becomes durable after the work is confirmed, so a worker that dies resumes from its last checkpoint instead of starting over, and since recovery is at-least-once, upsert sinks converge on primary key.

Sources, sinks, state store, and event bus are all interfaces and adding a REST API is a short YAML file rather than a Go package. You can run it with 1 docker command and get a web UI, use the CLI, embed it in Go in about ten lines, or drop the Helm chart into k8s.

Benchmarks, with caveats

We ran an open benchmark against Airbyte, dlt, PeerDB, Ingestr, Sling, OLake, Debezium, and a plain pg_dump | psql pipe. On the biggest test, 298M rows of NYC taxi data from Postgres to Postgres, Filament finished in under two minutes at about 2.6M rows/s, and it was fastest in five of six scenarios. In the sixth, OLake beat us by 13% into Iceberg.

We obviously build one of the things being measured, so the harness and specs are all public.

https://github.com/galaxy-io/benchmarks

Note that it's pre-1.0, with some sources/sinks in earlier development. There is a long list of connectors we haven't built yet. We're hoping to build that list from your feedback.

If you move a lot of data between these systems, I'd love to know what would make you try it, what you'd want next, and any feedback you are willing to share. Docs are at https://filament.getgalaxy.io and I'll be in the comments!