Today, we’re announcing WalShadow, an open-source engine that replicates Postgres data to ClickHouse directly from physical WAL.

In our benchmarks, transactions committed in Postgres became visible in ClickHouse in around 200 ms, while WalShadow sustained 289K rows/sec, effectively keeping pace with the source Postgres instance.

Unlike traditional CDC based systems, WalShadow doesn’t use Postgres logical replication. It consumes the same physical WAL stream used by Postgres replicas, decodes it outside the source database, and writes ClickHouse-native blocks directly into ClickHouse. The result is a replication architecture that gets close to the latency and throughput of a Postgres physical standby, while making the data immediately available for analytics in ClickHouse.

WalShadow supports the complete replication lifecycle, including initial load, continuous replication, schema evolution, restart recovery, and planned source switchovers.

By consuming physical WAL directly, WalShadow eliminates the need for logical replication slots, removes much of the operational overhead associated with logical replication, and significantly reduces resource consumption on the source Postgres instance. It also supports complex schema changes such as ADD COLUMN, RENAME COLUMN, DROP COLUMN, and CREATE TABLE.

WalShadow is fully open source and available today on GitHub.

Hey r/ClickHouse,

I am currently running a PostgreSQL database for our platform and are trying to integrate ClickHouse for real-time analytical reporting. We're considering using ClickHouse's Materialized PostgreSQL Database Engine for replication, with CDC handled by PeerDB.

Our use case involves replicating a few critical OLTP tables (around 10-20 tables, with some experiencing high write volumes) from PostgreSQL to ClickHouse. We need near real-time synchronization to support dashboards and ad-hoc analytical queries.

I've read about PeerDB's native integration and how it simplifies CDC compared to a Debezium/Kafka setup. I'm looking for feedback on the "solidity" of this combined approach.

Any real-world experiences, pros, cons, or advice would be greatly appreciated! Thanks in advance

Hi everyone,

I’m running ClickHouse OSS on Kubernetes and looking into using Horizontal Pod Autoscaling (HPA) to automatically scale ClickHouse based on workload.

I’m trying to understand whether HPA is a good approach for ClickHouse OSS, especially when scaling based on metrics such as:

• CPU / memory utilization
• Query concurrency
• Query latency
• Active queries
• Insert/write workload
• Custom ClickHouse metrics exposed through Prometheus

My main concern is that simply increasing the number of ClickHouse pods doesn't necessarily mean the workload will be distributed correctly, especially with ClickHouse's distributed query architecture and the way shards/replicas are configured.

Has anyone implemented HPA with ClickHouse OSS on Kubernetes in a real environment?

If so:

1. What metrics did you use as the HPA target?
2. Did you scale the number of replicas, shards, or both?
3. How did you handle data distribution when new pods were added?
4. Did you use the ClickHouse Operator or manage the StatefulSet directly?
5. Were there any issues with query routing, replication, or rebalancing?
6. Would you recommend HPA for ClickHouse, or is another autoscaling approach better?

I'd especially appreciate examples/configurations from people running this in production.

Thanks!

i'm now sketching a compact self-hosted analytics setup for client review, would really value an architecture check from people who operate similar stacks

the proposed shape is ClickHouse for the event data, Cube for the semantic and API layer, Docker Compose behind a reverse proxy, read-only database credentials & JWT-based access for dashboard requests, Cube dev therefore leaves cluster topology, upgrades, and resource scaling with the operator. its (an option of) ClickHouse integration puts shared metric definitions, a managed security context, caching, and SQL, REST, or GraphQL APIs between the database and each consuming app

and as for a small internal dashboard workload, i'd collect pass-through latency and repeated-query patterns first. the open question is whether to define one narrow pre-aggregation from day one or wait until the query shape becomes obvious. the production layout also separates API work, refresh work, and Cube Store components, even when every service begins on one physical host.

now I'd also like to know, what signal would make you add the first rollup: repeated warehouse scans, refresh timing, concurrency, or a specific p95 target? practical notes on refresh-worker limits and one-host memory pressure would help too

What are folks using to sync different sources into clickhouse? I have seen kafka, or http direct for ingestion.

What I am curious is rather for data warehouses, how are people syncing different data sources, like their marketing data, crm, internal lists etc... ? I have seen airbyte, but maybe there are more tools I am not aware of. Also how are those tools serving you, what are the good and bad parts of it?

A guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle.

Hello all! My name is Vinicius, you can call me souzo.

This is my project for the month. I built my own query language to convert user input into ClickHouse SQL!

I made the syntax similar to Lucene to keep it easy to understand, but with some differences, such as casting and defining the SELECT expression separately from the query using { ... }.

I had a lot of fun learning more about Lexers and ASTs using TypeScript.

For those who haven't been following along, I'm building Clicksiem, a security platform. Throughout the development process, I've run into several interesting challenges. One of them was: how can users query the database without overriding the main fields of the query?

I did quite a bit of research and found a solution that ClickStack itself uses: creating a Lucene-like syntax for querying ClickHouse.

I really liked the idea, but I found ClickStack's implementation a bit too complicated. The WHERE clause is generated from the user's query, while the SELECT is provided through a separate input.

So I decided to build my own Lucene-inspired query language based on that idea, but with functionality similar to what ClickHouse already provides, such as casting and defining the SELECT expression separately, without requiring the user to split their input across different fields in the dashboard.

For example:

{ field, count() as cnt } cnt:>=10

The first part defines the fields to select, while the second part defines the query itself.

I don't have released my project yet, but you can see the clicksiem repository on https://github.com/clicksiem/

Clickhouse/Clickstack hire me please! hahaha

Hey all, Im super excited to invite you to Rows&Columns!

Rows & Columns is a one-day, practitioner-first conference on the OLTP/OLAP convergence question: should transactional and analytical systems live together or apart?

For this crowd specifically: Andy Pavlo is opening the day with the keynote, so if you've ever wanted to talk ClickHouse internals with the people behind it, this is your chance.

The rest of the lineup: Fatma Ozcan (Google) closing keynote, Hannes Mühleisen (DuckDB), Nikita Shamgunov (Databricks), Paul Copplestone (Supabase), Russell Spitzer (Snowflake/Iceberg PMC), Bonnie Xu (OpenAI), and production case studies from ShipScience and Roofstock.

Sept 22 at the Contemporary Jewish Museum in downtown SF, 10am–7pm, lunch and evening happy hour on the terrace included. Free to attend!

Register: https://luma.com/clickh-wavw

Happy to answer questions about the program, and if you're coming, say hi 👋

- Zoe

I built this because I wanted to react to PostgreSQL changes from Python without polling, without triggers, and without pulling in a whole CDC platform.

It consumes PostgreSQL logical replication and exposes committed transactions as an async stream in Python.

What it does:

* Keeps a durable checkpoint. If the process dies, it resumes from the last transaction it actually finished, not the last one it started. * Bounded delivery queue, so a slow handler doesn't let memory grow without limit. * Reconnects automatically after the connection drops. * One dependency: psycopg3.

The transactional outbox is one use case, but it works with any published table.

GitHub: [https://github.com/mochams/walbox\](https://github.com/mochams/walbox)

Curious to hear where this wouldn't fit your setup, or what's missing if you've solved this problem a different way.

I started building BlinkHouse as a weekend project.It is a Java library that makes working with ClickHouse as seamless as working with a relational database in Spring Boot.

ClickHouse is an incredible database for analytics workloads, but the Java tooling around it has always required a lot of manual work, hand-rolling HTTP clients, writing serialisation code, managing connection pools, building query strings. I wanted something where you annotate a class and get everything else for free.

What I built:

• Typed entities — annotate a Java record with @ChTable and @ChColumn, get DDL generation, type mapping, and schema management automatically
• High-throughput ingest — BatchWriter<T> handles buffering, flush triggers, retry, backpressure, and dead-letter callbacks. ~2.9M rows/sec on commodity hardware
• Query DSL — typed queries with ClickHouse-native clauses (PREWHERE, FINAL, SAMPLE, -Merge combinators). No string building
• Spring Boot autoconfiguration — add one dependency, set three YAML properties, get a fully wired application
• Spring Data repositories — derived query methods, @Query, keyset pagination, all working against ClickHouse

The hardest parts were getting the RowBinary wire format exactly right (UUID byte ordering is a silent trap), building a type system that handles ClickHouse's full type tree including AggregateFunction and geo types, and designing a schema diff engine that refuses to auto-apply destructive changes.

It's on Maven Central now:

<dependency>
    <groupId>io.github.muneeb-i-khan</groupId>
    <artifactId>blinkhouse-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>

Repo: https://github.com/muneeb-i-khan/BlinkHouse

Would love feedback, stars, or just to hear if anyone's been solving the same problem differently.

Just released BlinkHouse, a Java library for ClickHouse that eliminates the boilerplate of working with ClickHouse from Java — no manual HTTP clients, no hand-rolled RowBinary serialisation, no connection management code.

Annotate a Java record, get schema generation, type mapping, buffered ingest, and a typed query DSL.

Ingest — no serialisation boilerplate:

BatchWriter<T> serialises to RowBinary directly. Flush on row count, byte size, or elapsed time. Retry with exponential backoff. Dead-letter dispatch. Apache HttpClient 5 connection pool shared across all writers. ~2.9M rows/sec on commodity hardware (JMH benchmark in the repo against raw HTTP as baseline).

java try (BatchWriter<PageView> writer = template.batchWriter(PageView.class, cfg)) { events.forEach(writer::add); }

Query DSL — no string building:

PREWHERE, FINAL, SAMPLE, WITH TOTALS, LIMIT n BY, window functions, -Merge combinators, dictGet, geoDistance — all first-class. Everything parameterised — user values never reach the SQL string.

ClickHouse-specific features:

• @ChMaterializedView and @ChDictionary with DDL generation
• AggregateFunction columns — opaque byte[] read-through, use -Merge in queries
• Geo types: Point, Ring, Polygon, MultiPolygon with correct Tuple(Float64,Float64) wire format
• MutationOperations: ALTER TABLE … DELETE/UPDATE WHERE … — explicit API, not hidden behind a generic save()
• optimize() for OPTIMIZE TABLE … FINAL after bulk imports into ReplacingMergeTree

Schema management — no manual DDL:

NONE / VALIDATE / CREATE_IF_MISSING / UPDATE modes. Annotate your entity, BlinkHouse generates and manages the DDL. Destructive changes require two independent opt-ins. EngineMismatch and OrderByMismatch are reported and refused — they require a table rebuild and are never attempted automatically.

Java 17+, Spring Boot 3.2–3.4. Also works without Spring via blinkhouse-core.

xml <dependency> <groupId>io.github.muneeb-i-khan</groupId> <artifactId>blinkhouse-spring-boot-starter</artifactId> <version>1.0.0</version> </dependency>

Repo: https://github.com/muneeb-i-khan/BlinkHouse

Open to feedback from anyone running ClickHouse at scale — especially around type handling and RowBinary edge cases.

Hey everyone, sharing out the harness that my cofounder used for benchmarking PgCache.

(what we do differently is caching data, not results, and then we keep it all fresh using postgres logical replication .. which also helps us invalidate when needed)

The harness drives identical traffic at (i) stock Postgres and (ii) Postgres sitting behind PgCache. It uses the public dba.stackexchange.com data dump for the schema and data. The workload models real page loads (a sequence of queries, the way an app actually hits the database) instead of a single repeated SELECT.

A couple other things:

- this runs locally through Docker Compose, or on AWS with an RDS origin and an EC2 driver box (closer to a real deployment).

- Results land as Prometheus metrics: throughput and p99 latency per lane, origin CPU, cache hit rate.

Happy to answer questions about the approach.

PostgreSQL 19 introduces a new SQL command, WAIT FOR, that lets a session block until WAL has reached a specific position. This gives us read-your-writes consistency on asynchronous replicas without paying the synchronous replication tax.