Why We Didn't Converge: ClickHouse's VLDB Paper and the Architecture Agents Actually Need

The moment ClickHouse writes CPU code for your query

You run SELECT category, COUNT(*) FROM events GROUP BY category against 100 million rows. On most databases, the engine walks a bytecode interpreter row by row, dispatching through a switch statement for every tuple. ClickHouse does something else. It takes your specific aggregation, hands it to LLVM, and generates native x86-64 instructions for this exact query. Then it runs them.

The difference is 2 seconds versus 12 seconds. Same hardware, same data, same SQL. Six times faster, because the CPU is executing code written for this GROUP BY, not code written to handle any possible GROUP BY.

The ClickHouse team published their first VLDB paper on April 14, 2026, titled "Lightning Fast Analytics for Everyone." Buried in section 4 is a detail that reframes a decade of analytical-database design: JIT compilation for aggregations was in the first commit in 2016. Not added later as an optimization. Not a recent flex. It was there on day one, because the founders believed interpreters were the bottleneck and compilers were the fix.

This post is about what that paper reveals, why Snowflake and Databricks quietly walked away from true HTAP, why AI agents are spawning 500+ database branches in Lakebase, and how I'd actually design a data platform in 2026.

The 5-minute skim

What the VLDB paper reveals: ClickHouse is not just "fast Postgres." It is four decisions stacked: LSM-style MergeTree storage, vectorized execution on batches (not rows), LLVM JIT for GROUP BY and multi-key sort, and 90+ file format integrations. Remove any one and the performance story collapses.

Default recommendation: If you are building analytics today and already have an OLTP system, do not converge. Split. Send CDC from Postgres into ClickHouse. This is what Snowflake + Databricks + CockroachDB have all effectively endorsed by abandoning HTAP.

Where this breaks: Sub-second freshness with strict transactional consistency across OLTP and OLAP. If an AI agent needs to read an uncommitted order from the last 50 milliseconds and aggregate it against 3 years of history in the same query, composable struggles. That is where Oracle Unified Memory Core, TiDB HTAP+vector, and Databricks Lakebase are betting.

Key trade-off: Composable wins on cost, flexibility, and scale. Converged wins on latency and developer experience for agent workloads. Pick based on whether your consumers are humans or agents.

Why is this the week to talk about data architecture?

Four things landed within seven days and they tell one story.

April 14 — ClickHouse VLDB paper. The first peer-reviewed publication of the internals. Not a blog post. A 12-page VLDB paper with benchmarks, design rationale, and the admission that most of what makes ClickHouse fast was decided in 2016.

April 7 — ClickHouse 26.3 release. 27 features, 40 performance optimizations. Async inserts are now the default. JOIN reordering extended to ANTI, SEMI, and FULL joins. Sharded Map Storage gives 2-49x lookup speedup. Materialized CTEs are real. And WebAssembly UDFs via Wasmtime, which means you can write user-defined functions in Rust or Go and ship them as Wasm.

April 2026 — Databricks Lakebase GA follow-up. Lakebase hit GA in February 2026. By April, the blog post that matters is the one about database branching. AI coding agents are creating 4x more databases than humans. Average production branch depth is 10. Some teams run 500+. Every pull request gets its own isolated Postgres instance with copy-on-write storage.

April 2026 — "Data Lakehouse Architecture 2026." The Medium piece that crystallized the hot/warm/cold pattern. RisingWave materialized views for millisecond freshness, Iceberg for 30-60 second warm tier, Iceberg for cold historical. Kafka topics and Iceberg tables are converging into the same object via StreamNative's Lakestream.

The through-line: the industry stopped pretending one database does everything, and started designing for the fact that agents, not humans, are now the dominant query generator.

What are the four pillars of ClickHouse?

The VLDB paper is organized around four layers. I will keep each brief because the depth is in the paper itself.

1. LSM-style MergeTree storage. Data lands as immutable sorted parts. Background merges compact them. Primary keys are sparse (one entry per 8192 rows by default), which keeps the index in memory even for trillion-row tables. Compression runs column-by-column, so a timestamp column with low cardinality compresses to a few bits per value.

2. Vectorized execution. ClickHouse does not process rows. It processes blocks of 65,536 values at a time. Every operator — filter, aggregate, join — is written to consume and emit these blocks. This means modern CPUs get to use SIMD instructions, branch predictors stay hot, and cache lines do not thrash. It is the difference between calling std::vector::push_back 100 million times and calling memcpy once.

3. JIT compilation via LLVM. This is the trick from the opening. For GROUP BY aggregations and multi-key sorts, ClickHouse emits LLVM IR, compiles it to native code, and caches the result. The payoff scales with aggregation complexity. Simple COUNT(*) sees 2-3x. Multi-column GROUP BY with expressions sees 6-10x. The 2s vs 12s number is from the paper's own benchmark on 100M rows.

4. The integration layer. 90+ file formats. Parquet (with ALP encoding now landing in Arrow 58.2), ORC, Avro, JSON, CSV, native formats from half a dozen other systems. S3, GCS, Azure Blob, HDFS, Kafka, RabbitMQ, Postgres CDC, MySQL CDC. The thesis is that analytics does not live in one system, so the engine must read from everywhere. This is what lets you point ClickHouse at Iceberg tables today and Delta Lake tomorrow without migrating data.

Pull one pillar out and the story breaks. LSM without vectorization gives you a slow log-structured store. Vectorization without JIT gives you Presto. JIT without the integration layer gives you a fast system nobody can feed. The VLDB paper's argument is that all four must coexist.

The agent queries both sides. It hits Postgres for "what is the current state of order 1234" and ClickHouse for "how does this user's behavior compare to the last 90 days of cohort X." One reasoning loop, two stores. That is the composable pattern.

Why did Snowflake and Databricks pivot away from HTAP?

Five years ago the pitch was "one database for everything." Snowflake would handle analytics and operations. Databricks would be the lakehouse that also ran transactions. Both companies quietly walked back that claim.

Snowflake launched Unistore in 2022 and has since de-emphasized it. The Snowflake 2026 narrative is openly about Iceberg interop and letting customers use external engines. They figured out that analytical workloads and transactional workloads want different physical layouts, different consistency models, and different resource profiles. Trying to serve both from one engine means serving both badly.

Databricks shipped Lakebase — and Lakebase is Postgres. Not a columnar engine pretending to be transactional. A real Postgres fork with copy-on-write storage and branching. The Databricks message is now: use the lakehouse for analytics, use Lakebase for OLTP, and let Unity Catalog bridge them. That is composable, not converged.

The pattern that won: Postgres → CDC or ClickPipes → ClickHouse. CockroachDB made this official with their April 2026 ClickHouse webinar, where the recap explicitly endorses the split architecture for agentic AI workloads. The reason is physics. A row-store with MVCC and a column-store with LSM merges cannot share a storage engine without one of them being worse at its job.

What did the fintech learn the hard way?

A fintech I worked with in 2024 tried to skip this lesson. They built what they called a "unified platform" on Postgres — transactions and analytics in the same database, because "we will deal with scale when we get there."

They got there. By early 2024 they were processing billions of events per day. The analytics team wrote a dashboard query that did a seven-way join across orders, users, merchants, and three audit tables. It took 45 seconds. During those 45 seconds, the query held read locks on the orders table. Order processing — the actual revenue-generating path — slowed down. At peak hours, orders were queueing for 200ms, then 800ms, then timing out.

They tried the usual escape hatches. Partitioning orders by date — still row locks. Materialized views — 30-minute refresh intervals, which meant the dashboard showed stale data. Read replicas — replication lag drifted to 2+ hours during heavy analytical queries because the replica was saturated applying WAL.

They split. Postgres stayed the OLTP store. Debezium captured CDC into Kafka. ClickHouse consumed Kafka and materialized the analytical model. Three weeks of engineering.

The numbers after the split:

• Analytics query: 45s → 800ms (56x faster)
• Order processing P99: back to 40ms
• Storage cost: dropped, because ClickHouse compressed 6 months of analytical data into less disk than 2 weeks of raw Postgres tables took. Typical compression ratios were 8-12x for event data with repeated categorical columns.

The lesson: converging OLTP and OLAP in one engine is seductive because it looks simpler. The simplicity is a loan. You pay it back with interest the first time analytics and transactions fight for the same locks.

Is database branching really Git for data?

Databricks Lakebase shipped database branching in February 2026, and by April the usage data is striking. Their own post reports that AI coding agents are creating 4x more databases than human developers. Average team has 10 active branches. Some production setups run 500+ branches deep.

Here is why this matters. When a human opens a PR, they usually test against a shared dev database or a seeded fixture. When an AI agent opens a PR — and agents now open dozens per day per engineer — it needs isolation. Two agents running migrations against the same database will step on each other. So every PR gets its own branch. Copy-on-write means the branch is cheap: it shares pages with the parent until you write, then only the diffs are stored.

This changes the dev workflow in three ways:

1. CI becomes stateful. Your test database is not reset between runs. It is forked from production (scrubbed), mutated during tests, and discarded. Bugs that only manifest against real data shapes surface earlier.

2. Migrations get tested for real. You run the migration against a branch that looks like production. If it locks tables for 20 minutes, you see it in CI, not at 3am.

3. Rollback is instant. A bad deploy? Fork the pre-deploy branch and point the app at it. You do not restore from backup. You switch a pointer.

The 500+ depth number is the one that stopped me. That is an agent spawning branches of branches of branches, each representing a hypothesis it is testing. It is a different shape of computation than humans do, and it is what infra has to support now.

What is the Lakehouse 2026 pattern?

Three tiers, each with a clear job.

Hot tier: RisingWave materialized views. Millisecond freshness. Streaming SQL against Kafka or Pulsar topics. You define a materialized view; it updates incrementally as events land. Query latency is sub-100ms. Use this for dashboards that must be live and for agent loops that react to events in real time.

Warm tier: Iceberg with streaming writes. 30-60 second freshness. This is where Kafka topics and Iceberg tables are merging. StreamNative's Lakestream treats them as one object — you produce to Kafka, you query Iceberg. Equinox, Flink, or RisingWave handle the conversion. This tier is for "recent but not real-time" — last hour of orders, last day of sessions.

Cold tier: Iceberg historical. Partitioned, compacted, cheap. Years of history. Query engines (Trino, Spark, ClickHouse, DuckDB) all read the same Iceberg tables. Storage cost dominates and it is S3-cheap.

The reason Iceberg is eating Delta Lake's lunch for streaming workloads comes down to partition evolution. In Delta, changing a partition scheme requires rewriting metadata. In Iceberg, partition evolution is first-class — you evolve the spec and old data keeps its old partitioning while new data uses the new. For streaming systems where you might shard by minute and then later shard by hour, this is the difference between "we migrate over a weekend" and "we do not migrate at all."

The other Iceberg advantage is multi-engine. Delta is Spark-native — other engines support it, but Spark is the reference. Iceberg was vendor-neutral from day one: AWS, Google, Snowflake, Dremio, and ClickHouse all treat it as a first-class citizen.

Delta still wins on one thing: change data feed. Delta CDF is mature; Iceberg's equivalent (incremental reads) is less battle-tested. If your use case is "give me exactly the changes since version N," Delta is still the safer choice.

How should I think about the trade-offs?

Three live debates, in prose because tables lie about nuance.

Composable versus converged. Composable is Postgres plus ClickHouse plus CDC. Converged is Oracle Unified Memory Core, TiDB HTAP+vector, or Databricks Lakebase. Composable wins on cost (each engine does one job well), on scale (you can shard them independently), and on vendor choice. Converged wins on latency for agent workloads that need to correlate fresh OLTP state with historical OLAP in one query, and on operational simplicity (one system to run, not three). My rule: if your primary consumer is humans writing dashboards, go composable. If it is agents making decisions, evaluate converged — but benchmark first.

ClickHouse versus Snowflake. ClickHouse is open-source, self-hostable, and its cost at petabyte scale is an order of magnitude below Snowflake. Snowflake is managed, has better SLOs out of the box, has deeper integrations with BI tools, and does not require you to run compactions or worry about merge pressure. If you have a small data team and a lot of budget, Snowflake. If you have a strong infra team and a lot of data, ClickHouse.

Iceberg versus Delta Lake. Iceberg wins on partition evolution, multi-engine support, and vendor neutrality. Delta wins on change data feed and Spark-native optimizations. Both are converging — Delta is adding Iceberg compat, Iceberg is improving CDC. If you are starting today with streaming writes, pick Iceberg. If you are deep in the Databricks ecosystem, stay on Delta. Do not try to mix them in one table.

When should I split and when should I converge?

Split (composable) if:

• Your analytical queries run longer than 5 seconds on your OLTP store.
• You are seeing lock contention between analytics and transactions.
• Your storage cost is dominated by analytical data retention.
• You have more than one analytical engine in the picture (BI tool + ML training + ad-hoc).
• Your dev team is comfortable running CDC and a second data store.

Converge (HTAP-ish) if:

• Your agents need sub-100ms correlation between fresh writes and historical aggregates.
• Your data volume is low enough that one engine fits.
• Your ops team is small and cannot run two stores.
• You have strict transactional requirements across analytical reads (rare but real in finance and healthcare).

The honest answer for most teams in 2026 is split. The composable stack is mature. CDC tooling (Debezium, Fivetran, ClickPipes) is boring-reliable. ClickHouse is open-source and fast. Iceberg is vendor-neutral. The convergence story is real but it is still early — Lakebase is GA but young, Oracle Unified Memory Core is new, TiDB's vector integration is evolving.

Five things to take away

1. ClickHouse is fast because of four decisions, not one. LSM storage, vectorized execution, LLVM JIT, and 90+ integrations. Read the VLDB paper before you build your own.

2. Do not converge OLTP and OLAP in 2026. Snowflake and Databricks walked away from HTAP for a reason. The fintech war story — 45s to 800ms after splitting — repeats in every company that tries.

3. Postgres → CDC → ClickHouse is the boring-reliable pattern. Debezium, ClickPipes, or Fivetran for the pipe. ClickHouse for analytics. Postgres for transactions. This works at every scale I have seen.

4. Database branching changes CI. If your team uses AI coding agents, Lakebase or Neon-style branching is no longer optional. Budget for 10 branches per engineer and plan for depth.

5. Pick Iceberg over Delta for new streaming workloads. Partition evolution and vendor neutrality are the two features you will need in year three. Delta keeps its edge only if you are all-in on Databricks.