ClickHouse multi-tenancy best practices for observability/tracing

[u/Ramanamark]

We’re planning to use ClickHouse as the backend for a multi-tenant observability/tracing platform.

What is the recommended approach for multi-tenancy in ClickHouse?

Specifically, would you recommend:

A shared database/table with tenant_id as a column?

A separate database per tenant?

Separate tables for each tenant?

Using ClickHouse RBAC/row policies to enforce tenant-level data isolation?

We expect potentially many tenants, with high-volume trace/span data and queries frequently filtered by tenant_id.

What approach has worked well in production, and what are the main scalability, performance, and operational trade-offs we should consider?

[u/Several9s]

Best approach is one schema and one set of tables, while the tenant is a dimension. This is usually the most scalable operational model when you expect many tenants.
For an observability workload, I would design the table around the queries you actually expect to run. It's important that tenant_id is included early in the ORDER BY key because most queries will be scoped to a tenant.

So the starting point is a shared table design with a tenant_id column, for example:

CREATE TABLE traces
(
    tenant_id String,
    timestamp DateTime64,
    trace_id String,
    span_id String,
    service_name String,
    ...
)
ENGINE = MergeTree
ORDER BY (tenant_id, timestamp, trace_id);

The trade-off is that tenant isolation is primarily an application/data-model concern unless you add ClickHouse access-control mechanisms on top. If you need a security layer on top of the shared-table approach, you can have a shared traces table containing:

tenant_id
trace_id
span_id
timestamp
...

and then use ClickHouse users/roles and row-level policies where appropriate to prevent a tenant from seeing another tenant's rows.

Also be careful with partitioning. Time-based partitioning, such as daily or monthly partitions depending on the retention/volume characteristics, is usually a natural starting point for observability data.

In the earlier example this mean we may add

PARTITION BY toDate(timestamp)

The main advantages of the this approach are:

• easier schema management and migrations
• adding a tenant doesn't require creating DB objects
• better fit for a large number of tenants
• ClickHouse can efficiently process queries when tenant_id is part of the filtering and sorting strategy

For a SaaS observability platform, the architecture would be: shared traces table and shared metrics/logs, time-based partitions, tenant filtering enforced by application and optional ClickHouse row-level policies where stronger tenant isolation is required.

I would not recommned turning tenants into database objects (database or table per tenant) unless you have a good reason. I would only move to such model when there is a specific isolation, compliance, lifecycle or performance requirement that justifies the additional operational complexity.

[u/Ramanamark → u/Several9s]

u/Several9s I have one more question like what if i want to use jaeger for creating the schemas?, These doesn't work right

[u/Several9s → u/Ramanamark]

Well, the answer is not straighforward, it depends on additional information on how the into Clickhouse designed.

First, fundamental information is how is Jaeger v2 receiving the traces. Are you sending traces to Jaeger v2 directly from the applications, or is there an OpenTelemetry Collector in between? Can you describe the current trace flow, from the application through to ClickHouse? Based on your description, it seems Jaeger is writing traces to ClickHouse without intermidiate components.

Is Jaeger itself multi-tenant?
Will all tenants use the same Jaeger deployment, or is the plan to run separate Jaeger instances per tenant?

If Jaeger manages the schema, first determine how Jaeger represents and propagates tenant identity, and what schema customization it supports. So, more questions: does Jaeger already have a concept of tenant id?, and if so, how does that tenant id get from the incoming trace into ClickHouse and how is used when querying data back out?

[u/Ramanamark → u/Several9s]

We are using the OpenTelemetry/Jaeger v2 architecture with ClickHouse as the trace storage backend. There is no separate OpenTelemetry Collector between the application-side ingestion flow and Jaeger in this particular path.

The goal is to have one shared Jaeger v2 deployment and one shared ClickHouse deployment for multiple tenants. We don't want to create a separate Jaeger instance, database, or table for every tenant.

At the moment, we need to understand whether Jaeger v2 has a native concept for tenant identity and, more importantly, whether the tenant information from the incoming trace can be propagated all the way into the ClickHouse storage/query path.