I benchmarked 5 managed graph databases — and the "obvious" winner changed depending on what I measured

I benchmarked 5 managed graph databases — and the "obvious" winner changed depending on what I measured

The setup

I was asked to benchmark CognoDB Cloud — a new Neo4j-compatible managed graph database — against four other players in the space: Neo4j AuraDB, Memgraph Cloud, FalkorDB Cloud, and ArangoDB Oasis. Same dataset, same logical queries, same client machine, five free-tier cloud instances.

The dataset was SNAP's cit-HepTh citation network: 27,770 physics papers, 352,807 citation edges. Simple schema — (:Paper)-[:CITES]->(:Paper) — nothing fancy, which turned out to be exactly the point. A boring, well-understood graph is a good stress test, because there's nowhere for a slow query to hide behind data complexity.

I measured five things on every platform: how fast data loads in, how fast you can walk 1/2/3 hops from a node, how fast you can look something up by key, how fast you can aggregate across the whole graph, and how the database holds up under concurrent load. Full methodology, raw numbers, and every caveat are in the repo — this post is the story of what surprised me.

The first surprise: the leaderboard wasn't stable

If you'd asked me after the first two tests which database was "best," I'd have said Memgraph, no contest. It was the fastest on 1-hop, 2-hop, and 3-hop traversals, and fastest again on point lookups and indexed lookups. Neo4j AuraDB was a close second on all of these. Everyone else — CognoDB, FalkorDB, ArangoDB — clustered noticeably higher, in the 140–200ms range where Memgraph and Neo4j were sitting comfortably under 90ms.

Case closed, right? Then I ran the aggregation query — count citations per paper, across the whole graph, top 20 — and the leaderboard didn't just shuffle. It flipped.

Neo4j — second place on every single-hop query — took the crown on the full-graph aggregation. And CognoDB and ArangoDB, which were merely "a bit slower" on lookups, fell off a cliff here: 1.8 seconds and over 4 seconds respectively, against everyone else's few hundred milliseconds.

The lesson: a single-query benchmark tells you almost nothing about how a database handles a full-scan workload. Point lookups and short traversals are cheap, indexable, and highly parallelizable across most engines' internals. Counting relationships across an entire 350k-edge graph is a completely different kind of work — closer to a batch job than a query — and it seems to expose real architectural differences that don't show up anywhere else.

The second surprise: concurrency broke one platform completely

I ran a mixed 80/20 read/write workload at two concurrency levels — 10 clients and 40 clients — for 10 seconds each, and measured total throughput.

Four of the five platforms behaved exactly like you'd hope: throughput scaled up roughly 3.6x to 4.1x when I quadrupled the client count, which is a healthy, expected result for a managed database designed to handle concurrent load.

ArangoDB scaled 1.05x. Fifteen ops/sec at 10 clients. Sixteen ops/sec at 40 clients. Quadrupling the concurrent load bought essentially nothing.

I didn't want to just report a scary number without checking myself first, so I ran db.explain() on the aggregation query to rule out the obvious explanation — a missed index. It wasn't that: the query plan confirmed the edge index was being used correctly, no planner warnings. Combined with the concurrency flatline, my best read is that ArangoDB's free-tier Oasis deployment is hitting some kind of connection or protocol ceiling under load — it's the only platform here using HTTP/REST instead of a lightweight binary protocol (Bolt for three of the others, native RESP for FalkorDB), and that difference seems to matter a lot more under concurrent pressure than it does for a single query in isolation.

I can't prove that's the root cause with certainty from a free-tier black box — but it's a testable hypothesis, and I'd rather hand you a good hypothesis with the receipts than a confident-sounding guess.

So how did CognoDB, the actual subject of this benchmark, do?

Mid-pack on traversals and lookups — slower than Memgraph and Neo4j, faster than FalkorDB and ArangoDB. It showed the same aggregation slowdown pattern as ArangoDB (1.8 seconds), though nowhere near as extreme.

The more interesting finding wasn't a number — it was a non-event. CognoDB speaks the Bolt protocol and Cypher, same as Neo4j. I wrote my connection code once, pointed it at Neo4j, and it worked. I changed three environment variables — URI, username, password — pointed the exact same code at CognoDB, and it worked too. No new driver, no new query syntax, no rewrite. That's not a performance number, but if you're a team already running on Neo4j and evaluating whether to try an alternative, "zero code changes to switch" is a genuinely practical thing to know, arguably more practical day-to-day than a few hundred milliseconds of aggregation latency.

CognoDB's dashboard was also the most transparent of the five, live node/relationship/storage counts front and center, no "upgrade to see your own metrics" wall — which, unglamorously, is also Neo4j AuraDB Free's biggest annoyance (its CPU/storage graphs are locked behind a paid tier).

What I'd flag before you trust any of these numbers

Two honest limits on this whole exercise, both documented in full in the repo:

The five databases weren't on equal hardware. Free tiers aren't a menu you can customize — CognoDB gave me 0.5 vCPU and 512MB RAM, Memgraph's only free option was a 2GB/2-CPU trial, FalkorDB's documented limit is a mere 100MB (that it clearly exceeded without complaint, which is its own small mystery). You're reading "how each vendor's free tier performs," not a resource-normalized shootout.

The five databases weren't in the same region. CognoDB and Neo4j happened to land in the same AWS region by coincidence of default settings; Memgraph was in Frankfurt, FalkorDB in ap-south-1. Some of what looks like "Memgraph is fast" might partly be "Memgraph's server is close to my laptop." I can't fully separate network geography from engine speed with this setup, and I'd be lying if I presented these numbers as pure, geography-free engine benchmarks.

Neither of these caveats invalidates the findings — the aggregation and concurrency results in particular are too large and too consistent to be explained by network noise alone — but they're the reason I'd treat this as a solid starting point for further testing, not a final verdict.

The full breakdown

Every metric, every query, every script, and the complete honest-caveats list (including the TLS certificate bug that took me an embarrassing amount of time to track down) is in the GitHub repo. It's fully scripted — npm run load:all && npm run bucket:all && npm run bench:all reproduces the entire thing from a clean free-tier account on all five platforms.

If you've benchmarked any of these five yourself and gotten different numbers, I'd genuinely like to hear about it — that's exactly the kind of thing that makes a benchmark like this useful instead of just a screenshot of some numbers.