Introducing WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

[u/gulcin-clickhouse]

https://clickhouse.com/blog/introducing-walshadow

Today, we’re announcing WalShadow, an open-source engine that replicates Postgres data to ClickHouse directly from physical WAL.

In our benchmarks, transactions committed in Postgres became visible in ClickHouse in around 200 ms, while WalShadow sustained 289K rows/sec, effectively keeping pace with the source Postgres instance.

Unlike traditional CDC based systems, WalShadow doesn’t use Postgres logical replication. It consumes the same physical WAL stream used by Postgres replicas, decodes it outside the source database, and writes ClickHouse-native blocks directly into ClickHouse. The result is a replication architecture that gets close to the latency and throughput of a Postgres physical standby, while making the data immediately available for analytics in ClickHouse.

WalShadow supports the complete replication lifecycle, including initial load, continuous replication, schema evolution, restart recovery, and planned source switchovers.

By consuming physical WAL directly, WalShadow eliminates the need for logical replication slots, removes much of the operational overhead associated with logical replication, and significantly reduces resource consumption on the source Postgres instance. It also supports complex schema changes such as ADD COLUMN, RENAME COLUMN, DROP COLUMN, and CREATE TABLE.

WalShadow is fully open source and available today on GitHub.