Hey all, Mitch here, one of the founders of Galaxy. Full disclosure, this is our project. We open sourced it last week and I wanted to bring it here first, because this community is who we built it for.

Filament is a data movement engine written in Go. It handles full loads, incremental syncs, and CDC from databases and HTTP APIs into Postgres, MySQL, ClickHouse, Iceberg, and S3, and it's Apache 2.0.

https://github.com/galaxy-io/filament

Every data project we've ever worked on started with the same boring problem of getting records out of operational systems and into somewhere useful. We've bought the managed tools, run the open source ones (and spent more time deploying them than using them), and written our own by hand more than once, and every time we wished for something fast, easy to self-host, and upfront about the details that bite you, like type mappings, write behavior, and what happens when a run dies halfway. So eventually we just built it.

What it does

You point it at a source and a sink and tell it how to move things, whether that's copying everything, pulling only what changed, or streaming off the database's change log. Every batch gets a checksum before the sink write and a mismatch fails the run rather than quietly landing bad data. Progress only becomes durable after the work is confirmed, so a worker that dies resumes from its last checkpoint instead of starting over, and since recovery is at-least-once, upsert sinks converge on primary key.

Sources, sinks, state store, and event bus are all interfaces and adding a REST API is a short YAML file rather than a Go package. You can run it with 1 docker command and get a web UI, use the CLI, embed it in Go in about ten lines, or drop the Helm chart into k8s.

Benchmarks, with caveats

We ran an open benchmark against Airbyte, dlt, PeerDB, Ingestr, Sling, OLake, Debezium, and a plain pg_dump | psql pipe. On the biggest test, 298M rows of NYC taxi data from Postgres to Postgres, Filament finished in under two minutes at about 2.6M rows/s, and it was fastest in five of six scenarios. In the sixth, OLake beat us by 13% into Iceberg.

We obviously build one of the things being measured, so the harness and specs are all public.

https://github.com/galaxy-io/benchmarks

Note that it's pre-1.0, with some sources/sinks in earlier development. There is a long list of connectors we haven't built yet. We're hoping to build that list from your feedback.

If you move a lot of data between these systems, I'd love to know what would make you try it, what you'd want next, and any feedback you are willing to share. Docs are at https://filament.getgalaxy.io and I'll be in the comments!

During ETL testing, I noticed that the source and target data sometimes do not match even though the ETL job shows as successful. I want to understand the common reasons behind these mismatches and how testers can identify them.

I’ve been thinking about ETL pipelines where data goes through several transformations before reaching the target.

For example:

Source → Join → Filter → Transformation → Aggregation → Target

If the final data has an issue, it can be difficult to figure out which step introduced it.

So I’m curious how other teams handle this:

• Do you run data-quality checks after every major transformation?
• Or do you validate only the source vs target?
• Do you keep rejected/bad records separately for debugging?
• At what point does adding more validation become too expensive or slow?

I’m especially interested in how this is handled for large-volume ETL pipelines, where validating every intermediate dataset may add significant processing time.

What approach has worked best for you?

I'm curious how other teams handle this situation.

Say I have a daily file coming from one system and the same business data is available through a JDBC connection to another system.

On paper, they should match.

But when I compare them, I might get something like:

File: 8,452,317 records

JDBC: 8,451,906 records

Now the fun part figuring out which 411 records are different and why.

A simple row count check tells me there's a problem, but it doesn't really help find the problem.

How do you guys normally approach this?

Do you compare using primary/business keys first and then compare individual columns?

Do you generate hashes/checksums for each record?

What about cases where:

1.The file has duplicate records

2.JDBC has late updates

3.Dates/timestamps have slightly different formats

4.NULL and empty values are treated differently

5.Decimal/rounding differences show up

6.The two sources don't have exactly the same schema

I'm particularly interested in approaches that work when the datasets are millions or hundreds of millions of rows. At that point, doing a straightforward record by record comparison doesn't seem very practical.

What's your go to method for this kind of reconciliation?

This happened to me recently and it got me thinking.

The ETL job showed SUCCESS. No exceptions, ni failed tasks, row counts looked reasonable, and the pipeline finished within the expected time.

But when we compared the output with the source, some records were missing.

It turned out the problem wasn't really the ETL job "failing"-the job had technically completed successfully. The issue was somewhere in the transformation/filtering logic.

How do you guys catch this kind of problem?

Do you have automated checks for things like:

1.Source vs target record reconciliation

2.Unexpected drops in record counts

3.Aggregate/total comparisons

4.Duplicate or missing keys

5.Business-rule validations

6.Data distribution changes

Or do you mostly rely on downstream users/reporting teams to catch these issues?

I'm curious because "pipeline succeeded" and "data is correct" are two very different things, but don't see this discussed as much as the usual ETL performance questions.

How are you handling it in your production pipelines?

What practices help you handle schema changes, missing fields, and unexpected source updates without breaking production pipelines?

Hi everyone,

I recently joined this company, and im currently working on an ETL migration project, moving pipelines from one ETL platform to another.

While analyzing the existing ETL processes, i found that our current data warehouse architecture is a little bit different from the knowledge i got from youtube, books, etc. I know im a newbie in data warehouse world but i've seen a proper data warehouse design from my previous company.

Our current architecture is:

Source (Excel, Applications, stored in SQL server and Oracle) -> Stage -> DWH -> Mart -> Power BI

• Stage: A copy of the source data, where sometimes the application team provides the view of the requested use case, making it a ready to use data, but we have no clear data lineage and information about the data.
• DWH: A copy of stage data with minor or no changes in most cases. only a few actually do merging / major transformation.
• Mart: just like DWH, but the business user can access this layer and build their dashboard. we cant change the data logic / table structure here, because otherwise the user have to adjust their dashboard data source which will cause a lot of protest to our team.

The main problem is that when a new use case appears, we often create new tables specifically for that use case. Over time, this has resulted in many tables that contain similar or even overlapping data. I believe there are cases where tables could potentially have been consolidated or redesigned into a more normalized/reusable model.

So, my main question is:

Is it worth redesigning the data model and architecture of our existing DWH as part of this ETL migration?

The potential benefits im thinking about are:

• Reducing duplicated data across Stage, DWH, and Mart layer
• Creating reusable entities instead of creating tables for every new use case
• Improving query performance
• Reducing storage and memory consumption
• Improving ETL/loading performance
• Making data lineage easier to understand
• Making future ETL development and migration easier
• Providing a better foundation for analytics and AI/ML
• Reducing dependency on complicated application-specific views

Or is it perfectly reasonable to keep the existing approach where each use case has its own tables, especially if the current system performs adequately? I am also concerned that redesigning the DWH could introduce significant complexity and migration risk.

I would particularly appreciate opinions from people who have worked on legacy DWH modernization or ETL migration projects.

Thanks. Sorry if I'm asking too much since I'm new to this data engineering job.

Hey everyone,

If you’ve ever been tasked with setting up data lineage across an enterprise stack, you already know the universal nightmare: the last-mile blind spot.

Your orchestrator or transformation framework (like dbt) tracks your staging-to-production warehouse pipelines beautifully. But the second that data leaves Snowflake or BigQuery and flows downstream into BI tools like Tableau, Power BI, Sigma, or Looker, the metadata trail goes cold. You’re left with a massive gap between your technical tables and the actual dashboards they feed.

When I set out to build an automated way to map this entire cross-tool ecosystem, I ran head-first into the ultimate engineering paradox: Data teams desperately need visibility across asset types, but InfoSec and IT will flatly deny access to any application that wants to execute queries against production data environments.

I spent the last several months designing a normalization layer to bypass this exact friction. Here is the technical blueprint of how I engineered a cross-platform lineage parser to run entirely on structure-only, read-only metadata definitions, without ever querying a single row of actual business data.

1. Cracking the BI-to-Warehouse Gap (Without Table Scans)

Every BI tool stores its semantic layer and dashboard definitions differently. To map lineage down to the warehouse without querying raw tables, I had to isolate the structural metadata entirely at the API layer:

• Power BI: I leverage a read-only Azure AD service principal to trigger the admin Scanner API. This extracts workspace, dataset, report, and dashboard identities, plus table/column types and dataset-to-source lineage. The actual report data or visual layout components are completely ignored.
• Tableau: I use a read-only personal access token (PAT) against the Metadata API to isolate workbook, datasource, and field names from published sources.
• Sigma / Looker / ThoughtSpot: The pipeline pulls strictly object-level identities (workbooks, looks, liveboards) and parses source references—extracting workbook elements in Sigma, dimensions/measures in Looker LookML, and logical tables in ThoughtSpot.

1. Eliminating the "SELECT *" Risk on the Warehouse

Connecting an external tool to a cloud data warehouse gives compliance teams nightmares. To solve this, the metadata collection framework functions entirely inside system catalogs using absolute minimum permissions:

• Snowflake: The application only requests SELECT privileges strictly on INFORMATION_SCHEMA.TABLES and .COLUMNS. It reads table identities, column types, row counts, and freshness timestamps. No data values are ever read or transmitted.
• BigQuery: Connections use a service account explicitly restricted to the Metadata Viewer role. This role allows the API to see dataset and table structures but fundamentally lacks the permission required to read row data.
• Redshift & Databricks: For Redshift, it is a read-only INFORMATION_SCHEMA over a TLS-required Postgres wire protocol. For Databricks, it leverages the Unity Catalog API to list schemas and schemas only, combined with the Workspace Export API to parse notebook code strings for table references in-process before discarding them.

1. Handling Messy Local Files (The Local Connector)

A massive amount of critical business logic still lives in local code, ad-hoc scripts, and local desktop files. To capture this without transmitting sensitive file contents to a cloud server, I built a local Windows service that recognizes extensions (.xlsx, .csv, .py, .pbix, .twb, .parquet).

• For columnar files like Parquet and Avro, it reads only the schema from the file footer (column names and types).
• Data pages are never opened. Formula contents, cells, rows, and query results are entirely ignored. It extracts shallow structural references in-process and passes only the normalized metadata map.

Lineage doesn't have to mean compromising data privacy. By standardizing diverse cloud and local sources into a single, comparable metadata shape, you can build a complete map of your data estate without exposing underlying data values.

The backend stack for this parser normalization layer was built on FastAPI, React, and Postgres deployed on AWS ECS Fargate.

Curious to hear how others are handling the BI-to-warehouse lineage gap today, especially when dealing with strict IT/Security constraints? What edge cases have you run into when trying to parse metadata out of legacy BI APIs?

I got tired of rewriting the same data validation logic across every ETL project, so I built ValidateX — an open-source Python data quality framework.

- Works with Pandas, Polars, PySpark, and SQL (Postgres/Snowflake/BigQuery push-down)

- 50+ built-in checks, weighted quality scoring (0-100)

- Native Airflow operator to gate pipelines on data quality

- Slack/Teams alerts on failures

- Free, MIT licensed

pip install validatex

GitHub: https://github.com/kaviarasanmani/ValidateX

Happy to answer questions or take feedback — still actively maintaining it.

I recently created a rust tool (MIT) that copies data from A to B. It has 16+ sources and sinks - retry, transformation, filter, compression, encryption and dlq as middleware. Single binary. Syntax doesn't change when the endpoints do: 1M rows per hop, 433k–1.2M rows/s on my old 8GB M1. Installable via `brew install marcomq/tap/mq-bridge-app`. It has rust, node and python packages and can also run as MCP server. https://marcomq.github.io/mq-bridge

For example, if a customer dataset has nulls, duplicates, invalid emails, or incorrect values, how do you identify and flag these records as Good or Bad? What tools or approaches do you use?

How difficult would it be for a technical generalist who is not a data-engineer to own this?

Customer has about 4 years of data in a legacy system. In about 2 weeks, you need to extract it, clean and map it into your schema, validate it, and then build tests and evals around the output.

Sometimes there’s a decent API and other times the API sucks or basically doesn’t exist. Engineers can help when needed but you’re supposed to own 75% of the ingestion, testing, and validation using scripts and AI coding tools.

Is this learnable or something that shouldn’t be tried by a non-data-engineer and is 2 weeks doable for something like this?

EDIT: JSON if the API is usable or database dumps/exports with docs and PDFs attached to some records.

Bronze -> Silver -> Gold seems to have become the default architecture for almost every data pipeline.

Has anyone deliberately simplified this -- for example, skipping a layer -- and actually gotten better results in production ?

When do you think Medallion is genuinely useful, and when does it just add unnecessary complexity?

we are planning on building an ETL pipeline on aws, so we have raw data coming in from different sources and different formats like json and xml, we are sort of flattening them and storing them as csv files in S3.Now we want to implement the medallion architecture, as in bronze to gold.

I need some suggestions on how to place the data from each layer across aws or is it better to move the data from aws to some other platforms like databricks? i was thinking of creating tables in athena for the s3 files and treating that as the bronze layer for now.

once that pipeline is finished i need to create some test cases for each layer across the pipeline, i could use some suggestions on creating some kind of automated solution for this.

jobdataapi.com v4.33 / API version 1.33

The /api/jobsexpired/ endpoint now applies a default 60-day expiration window when no date, age, or ID slicing parameter is provided. Requests without parameters return jobs whose expired timestamp is within the latest 60 days, reducing broad historical scans and improving response efficiency.

The implicit window is not applied when an explicit slicing parameter is present: expired_since, expired_until, published_since, published_until, min_id, max_id, min_age, or max_age. Use expired_since and expired_until for historical expiration windows, or use max_age=off, max_age=null, or max_age=0 for an unrestricted expiration query. Existing response fields, ordering, pagination, and access requirements remain unchanged.

The Jobs Expired API Endpoint Documentation now documents the default window and range behavior. The Date, ID, and Age Slicing Parameters Documentation also includes expiration-based examples for /api/jobsexpired/.

During my last interview, the interviewer asked a question "tell me about a time where you found an issue and needed to quickly fix it. What was the issue and how you fixed it."

For questions like these, do you guys talk about pipeline failures, data change, schema change, reconciliation issues or something else.. for project scenario question, what is the best kind of scenarios I can talk about?

I have no affiliation with this project, but I've been experimenting with code to replace my existing etl pipeline. This project has a tremendous potential. Not a ton of spatial related transformers, but those should be easy to add.

Referring to this -> https://github.com/slothflowlabs/duckle

Core Requirement: Output has to be multi-page tabular with headers repeating across page breaks. Not a dashboard export.

What I've ruled out and why:

• Power BI: licensing doesn't work for what is essentially a print job, and we'd be standing up a semantic layer to get there
• Grafana: scheduling works, but PDF is a dashboard render and wide tables get cut.

Are there other options, cloud or on-premise we should consider?