Most telemetry queries now come from software, not from people
Observability tooling was built on an assumption that held for a decade: a person is the last mile. Dashboards get rendered, metrics get rolled up into something an eye can parse, logs get sampled down to whatever an on-call engineer might plausibly read at 3am. A five-minute average is fine, because nobody was ever going to look closer than that.
That assumption has quietly stopped being true. A growing share of the queries hitting telemetry stores now come from software: automated remediation, fraud and risk scoring that runs continuously, capacity controllers, and the agentic services that ClickHouse put a product name to with ClickHouse Agents at Open House 2026. The trend is bigger than that one launch, but the launch is a decent marker for when it became obvious.
Software queries differently to people. It does not want the five-minute average, because the thing it is hunting for was averaged away. It wants the raw events, it wants them at high concurrency, and it wants them at three in the morning on a Sunday whether or not anyone is awake.
Legacy observability platforms are close to the worst possible fit for that. They keep costs down by sampling, which throws away the exact granularity the automation needs, and they charge by the ingested gigabyte, so "keep everything and query it often" becomes a procurement negotiation. Then the concurrency caps arrive and the ceiling turns out to be technical as well as financial.
ClickHouse is fast because of how it stores data
ClickHouse is an open-source, column-oriented database built for online analytical processing. The speed is not down to a clever optimiser. It comes from the storage layout, and knowing why matters, because the same layout is what makes it unsuitable for the workloads further down this page.
Each column is stored separately
Ask a row-oriented database for the 95th percentile latency of a service over the last hour and it reads whole rows off disk, then discards almost all of each one. Every field you did not ask for still got fetched.
ClickHouse stores each column separately. That same query touches three columns out of fifty and reads about three fiftieths of the data. At a few billion rows, that ratio decides whether you get an answer or a timeout.
Queries are processed in blocks and parallelised automatically
Once the data is columnar, ClickHouse can work through it in blocks instead of a row at a time, which keeps values in cache and lets it use SIMD instructions properly. Big queries parallelise across cores without you asking, and across shards without the application knowing. The documentation quotes hundreds of millions of rows per second per server. Vendor numbers usually deserve a squint, but this one holds up in production better than most.
Compression reduces both storage and compute cost
Columns of similar values compress hard, and ClickHouse ships codecs tuned for particular shapes of data, notably timestamps and numeric series that change slowly. Ten to one is unremarkable on telemetry. We have seen considerably better on metrics.
That pays twice over. Less data on disk is a storage bill you do not pay. Less data to read is a compute bill you do not pay. Push older partitions out to object storage and keep them queryable, and the same effect stretches across the whole retention window instead of just the hot data.
ClickHouse puts a number on the combination through CostBench, its own open benchmark, which has ClickHouse Cloud at 23x better read-side cost-performance than the nearest cloud data warehouse tested, and rather more once ingest and preparation are counted. Treat it as what it is, a vendor benchmark, though the workloads, pricing assumptions and code are all published, which is more than the competition has offered. The $250M ARR and 4,000 customers announced alongside it are a consequence of the arithmetic working, not a reason to adopt anything.
ClickHouse is the wrong database for four common workloads
Everything above is a trade. The design that makes analytical queries fast makes a different class of query worse than an ordinary database would manage. Most ClickHouse regret we get called in to unpick starts with someone skipping this bit.
| Use ClickHouse for | Why it fits |
|---|---|
| Observability, logs, metrics and traces at full fidelity | High-volume append-only events, queried by aggregation over time windows. Compression makes un-sampled retention affordable. |
| Clickstream, product and user-behaviour analytics | Wide event tables where any query touches a handful of columns, with high concurrency from dashboards and APIs. |
| Financial, market and risk data | Very high ingest rates with continuous analytical queries over recent and historical windows. |
| IoT, telemetry and time series | Ordered numeric series where specialised codecs and sparse indexing on time work in your favour. |
| Customer-facing analytics and machine-driven querying | Sub-second responses at concurrency, over raw events rather than pre-computed aggregates. |
| Workload | Documented constraint | Use instead |
|---|---|---|
| Transactional application state | ClickHouse has no full-fledged transactions. | PostgreSQL |
| Frequent updates and deletes of individual rows | ClickHouse cannot modify or delete already-inserted data at a high rate and low latency; mutations are asynchronous rewrites. | PostgreSQL, or an event-sourced model with ClickHouse as the sink |
| High-QPS single-row lookups by key | The sparse primary index makes point queries retrieving single rows inefficient. | Apache Cassandra or a key-value store |
| Full-text search and relevance ranking | Text search exists but is not the engine's purpose; relevance scoring and analysers are far weaker. | Elasticsearch |
| Low-volume reporting over small datasets | The operational overhead of a distributed cluster is not repaid below a certain scale. | PostgreSQL with appropriate indexes |
None of that is a defect list. It is the bill for the design, and the project documents it plainly enough. What goes wrong is adopting ClickHouse as a general-purpose database and finding out which half you are in once the data is already there.
One that catches people in particular: ALTER TABLE ... UPDATE exists, so it looks like an update. It is a mutation, which rewrites the affected parts in the background. Fine occasionally. Run it per user action and the cluster spends its life merging.
Production clusters are much harder than a proof of concept suggests
ClickHouse demos well. Point it at a representative dataset on a couple of nodes, run the queries that used to take a minute, and the room is generally convinced. That stage rarely fails.
The distance between that and a cluster you would put an on-call rota against is where projects tend to stall, and the reasons are operational.
ClickHouse has few safe defaults to fall back on
The decisions that matter at scale are the ones a proof of concept never forces you to make. The sort key on a MergeTree table governs what the sparse index can skip, so getting it wrong is not a tuning problem, it is a table rewrite. Shard and replica topology has to line up with how the data is keyed or every query fans out to every node and your concurrency headroom evaporates. Distributed DDL has to land consistently on a cluster that is still taking writes.
And ClickHouse Keeper has to stay healthy under ingest load, which is where the fun tends to start, because when Keeper is struggling the error you actually see is usually Too many parts somewhere else entirely, and the team spends a day looking at the wrong thing.
These are narrow skills, hard to hire and expensive to keep, and they tend to end up concentrated in one or two people. That becomes an availability risk as soon as one of them goes on holiday. Meanwhile the engineers hired to build product are managing partitions.
Kubernetes does not reduce the database work
Running ClickHouse on Kubernetes is still the right call for most platform teams. It just does not reduce the database work, and it adds some.
Topology now has to survive pods being rescheduled onto different nodes. Schema migrations have to run against a live sharded cluster with traffic on it. Storage tiering between local NVMe and object storage has to be designed on purpose, because that decision is simultaneously where most of the cost saving lives and where most of the ways to lose data quietly live.
The three options differ mainly in operational burden and lock-in
There are three ways to do this. Query latency will not separate them, since ClickHouse is fast whoever runs it. The differences are in where the operational burden ends up and how hard it is to leave.
| Dimension | Legacy SaaS observability | Self-managed ClickHouse | Managed ClickHouse (Digitalis) |
|---|---|---|---|
| Data fidelity and retention | Aggressive sampling; raw log granularity is discarded before you can query it. | Full-fidelity raw event retention, if you design the tiering for it. | Full-fidelity retention with tiering designed against your query and cost profile. |
| Concurrency and cost model | Per-gigabyte ingestion pricing, high per-query cost, hard concurrency caps. | Predictable compute cost and high concurrent throughput; sizing is on you. | Predictable compute cost, sized and re-sized to the workload as it changes. |
| Vendor independence | High lock-in to a proprietary ecosystem and its data format. | Fully cloud-agnostic, open-source control. | Open source throughout; multi-cloud, hybrid and bare-metal deployment. |
| Operational burden | Low to start, paid for in escalating licensing. | High and permanent: topology, MergeTree tuning, Keeper, migrations. | Carried by database SREs, with 24/7 cover and defined escalation. |
| Integration with the rest of the stack | An isolated telemetry silo with restricted access to the data. | Custom integration work for Kafka, PostgreSQL and Kubernetes. | Integrated as one data backbone alongside Kafka, PostgreSQL, Elasticsearch and Cassandra. |
Why Digitalis
We built our ClickHouse practice around one constraint: taking the operational load off a team must not hand them a new lock-in to replace the one they just escaped. That rules out a few otherwise convenient shortcuts, and it shapes what the service is.
The cluster stays in your account
We are a vendor-agnostic consultancy, and we push open source because it keeps our clients in control of their own technical assets. Clusters are deployed and run natively on AWS, Azure or GCP, or on bare-metal Kubernetes for the teams still buying their own hardware. It is standard open-source ClickHouse in your cloud account, not a fork and not a wrapper, so moving it later is a decision rather than a project.
ClickHouse needs the rest of the data platform around it
Read the two fit tables again and the conclusion is unavoidable: a fast analytical engine on its own does not get you very far, because four of the things your application needs are somebody else's job. ClickHouse earns its keep as the analytical layer of a platform, fed by Apache Kafka, sitting alongside PostgreSQL for state, Elasticsearch for search and Apache Cassandra for distributed operational workloads, all of it on Kubernetes.
Keeping an estate like that healthy needs monitoring that understands distributed databases, not a generic infrastructure dashboard that tells you CPU is high. We use AxonOps for that, so we are looking at cluster internals and getting warned before anything degrades.
What we take on:
- Cluster design: MergeTree settings, sort keys, shard and replica topology worked out from your query patterns, not lifted from a reference architecture.
- Deployment across hybrid, multi-cloud, on-premises and bare-metal Kubernetes, with portability intact.
- Zero-downtime upgrades, live schema migrations, storage tiering, and restores that have actually been tested.
- 24/7 database SRE cover for cluster health, failover and capacity, with named escalation.
- Integration with everything around it, so you end up with a data platform and not another silo.
The commercial case is predictable cost and engineering time returned
The bill becomes predictable. Per-gigabyte ingestion pricing goes away and you pay for compute you chose and control. Costs then track your architecture decisions instead of tracking however much data the business happened to generate last month, and keeping full-fidelity data stops being something you negotiate down every renewal.
Nobody owns your exit. The data, the pipelines and the infrastructure stay yours, in an open format, on whichever cloud you like or none of them.
Your engineers go back to building. This is usually the biggest number in the business case and the one that never makes it onto the slide, because it is the hardest to invoice: senior people stop doing partition management and go back to the work you hired them for.
Where to start
A short sequence beats a long evaluation.
- Work out what share of your analytical queries now come from software rather than from a person opening a dashboard. That ratio is the real trigger for all of this.
- Price your current platform at full fidelity with no sampling. It is the only honest comparison, and for a lot of teams the decision makes itself at that point.
- Check your access patterns against the two tables above before anything else. Frequent single-row updates or high-QPS key lookups mean you want a different database, and better to know now.
- Run the proof of concept, then work out the operating model before you plan the migration. That is the step everyone skips and the one that decides whether this lands.
- Decide deliberately whether running databases is a capability you want in-house. Buying it is a perfectly respectable answer. The bad outcome is drifting into doing it yourself without ever having made the decision.
For high-concurrency analytics over large volumes of events, the engine question is fairly well settled. Running it well is a separate problem, and a solvable one.
