Benchmarking 5 Graph Database Platforms on the Same 352k-Edge Graph: What Free Tiers Hide

Graph database benchmarks are easy to get wrong.

Most public benchmarks compare expensive production instances, use different datasets, ignore network latency, hide free-tier throttling, or quietly optimize one database more than the others.

For a recent engineering assignment from Wexa AI, I decided to do the opposite.

I built a small benchmark harness that compared CognoDB Cloud against four other graph database platforms using:

• the same dataset
• the same logical queries
• the same client machine
• free or entry-level resource tiers
• warm-up before measurement
• percentile-based latency reporting
• automated one-command runs
• honest caveats

The platforms tested were:

1. CognoDB Cloud
2. Neo4j Aura Free
3. Neo4j Sandbox
4. Docker-capped Neo4j Community
5. Memgraph Cloud

The dataset was the SNAP cit-HepTh citation network:

• 27,769 nodes
• 352,768 relationships
• papers citing other papers

Small enough to fit inside free tiers, but large enough to make the benchmark meaningful.

The full benchmark code, results, and documentation are available here:

GitHub repository: [Add your GitHub repository link]

The Goal Was Not to Find a “Winner”

The assignment was clear: the goal was not to crown a database.

The goal was to evaluate engineering rigor:

• fair methodology
• reproducible automation
• honest reporting
• clear analysis
• documented caveats

That matters because real-world database selection is messy.

There is no universal “best graph database.”

There is only the best database for:

• your data size
• your query patterns
• your latency requirements
• your operational constraints
• your budget
• your deployment model

So this benchmark was designed to measure behavior under constrained free-tier conditions, not to simulate an unlimited production environment.

The Fairness Rules

Before writing code, I set a few rules.

1. Same dataset everywhere

Every platform received the exact same graph.

2. Same logical queries everywhere

All platforms ran the same workloads, even where minor Cypher dialect adjustments were required.

3. Free or entry tiers only

No paid production instance was allowed to sneak into the comparison.

4. Small resource footprint

CognoDB’s free tier is intentionally small:

• burstable 0.5 vCPU
• 256 MB RAM
• 1 GB disk

So the other platforms were run on their free, trial, or capped self-hosted tiers.

5. Warm-up before measurement

Every read workload was warmed up first. Cold-start behavior was excluded from the main latency numbers and documented as a caveat.

6. Percentiles, not just averages

Averages can hide bad requests.

So the benchmark reported:

• p50 latency: typical behavior
• p95 latency: tail behavior

7. No hidden caveats

If something failed, throttled, crashed, or behaved strangely, it got documented.

That rule ended up being one of the most valuable parts of the project.

The Dataset

I used the SNAP cit-HepTh citation network.

It represents arXiv High Energy Physics Theory papers and their citations.

If paper A cites paper B, the graph contains a directed edge from A to B.

The final cleaned graph contained:

Each paper node had:

• id
• year
• field

The year property came from the original SNAP metadata.

The field property was generated deterministically for aggregation testing, so every platform received identical values.

This dataset was a good fit because it had enough relationships to make traversals and aggregations interesting, but it was still small enough to fit into free tiers.

The Benchmark Harness

I built the benchmark harness in C# using .NET 8 and the official Neo4j .NET driver.

For each platform, the harness did the following:

1. downloaded and prepared the dataset if needed
2. connected using secrets from environment variables
3. wiped old benchmark data
4. created indexes and constraints
5. loaded nodes and relationships in batches
6. ran warm-up queries
7. measured read workloads over 100 iterations
8. reported p50 and p95 latency
9. ran mixed read/write workloads at 1, 10, and 40 concurrent clients
10. saved results as JSON

Then a report generator converted the JSON files into Markdown tables.

The workflow was simple:

dotnet run -- cognodb
dotnet run -- aura
dotnet run -- sandbox
dotnet run -- memgraph
dotnet run -- docker

And then:

dotnet run -- report

No passwords were stored in the repository. All credentials were read from environment variables.

The Workloads

The benchmark measured six categories.

1. Ingest throughput

How fast can the platform load the graph?

Measured as:

• nodes per second
• relationships per second
• total wall-clock load time

2. Traversals

One-hop, two-hop, and three-hop queries from randomly selected start nodes.

Example:

MATCH (p:Paper {id: $id})-[:CITES]->()-[:CITES]->(q)
RETURN count(q) AS c;

3. Lookups

Point lookup by indexed id:

MATCH (p:Paper {id: $id})
RETURN p.year AS year;

Filtered lookup using indexed year:

MATCH (p:Paper)
WHERE p.year >= $y1 AND p.year <= $y2
RETURN count(p) AS c;

4. Aggregation

Group-by query over a node property:

MATCH (p:Paper)
RETURN p.field AS field, count(*) AS c
ORDER BY c DESC;

5. Mixed read/write workload

The mixed workload used:

• 80% reads
• 20% writes
• 1, 10, and 40 concurrent clients
• 30 seconds per concurrency level

6. Footprint

Where observable, the benchmark recorded:

• node count
• relationship count
• resource limits
• available platform metadata

Where a platform did not expose internals, the result was marked as not observable.

Results: Ingest Throughput

Memgraph dominated ingest.

Visual summary:

memgraph   14966 rels/sec  ████████████████████████████████
docker      8144 rels/sec  █████████████████
aura        7865 rels/sec  ████████████████
sandbox     2339 rels/sec  █████
cognodb     1209 rels/sec  ██

Memgraph’s in-memory architecture gave it a major advantage during batched loading.

Aura also performed strongly.

Docker Neo4j benefited from local execution and no cloud network overhead.

CognoDB and Sandbox were slower under the tested free-tier conditions.

Results: Traversal Latency

For traversal latency, Docker Neo4j had the lowest p50 numbers because it ran locally.

One pattern stood out.

For most platforms, latency barely increased from one hop to three hops.

For example:

• Aura: 146.2 ms to 146.8 ms
• Memgraph: 95.3 ms to 96.2 ms
• Sandbox: 697.8 ms to 698.0 ms
• CognoDB: 644.2 ms to 654.2 ms

That tells us something important.

At this dataset size, the actual traversal work was small compared with fixed overhead.

That overhead may include:

• network round-trip time
• request parsing
• query planning
• serialization
• connection handling
• free-tier CPU scheduling

In other words, the query engine may not have been the dominant cost. The request path was.

Results: Lookups and Aggregations

Lookup latency followed a similar pattern.

Point lookup p50

Aggregation p50

Docker’s low numbers again reflected its localhost advantage.

Among cloud platforms, Memgraph and Aura were the strongest in this test.

CognoDB remained stable, but its latency was dominated by what appeared to be fixed request overhead under the tested free-tier environment.

Results: Mixed Read/Write Workload

The mixed workload used 80% reads and 20% writes.

At 40 concurrent clients:

Visual summary:

docker     378.4 QPS  ████████████████████████████████
memgraph   374.4 QPS  ███████████████████████████████
aura       287.7 QPS  ████████████████████████
cognodb     61.2 QPS  █████
sandbox     51.3 QPS  ████

Docker and Memgraph achieved the highest throughput.

Aura scaled well and completed the workload with no recorded errors.

CognoDB and Sandbox showed lower absolute throughput, but both completed with zero recorded errors.

Memgraph produced five errors during the 40-client workload. I kept those errors in the results because hiding them would defeat the purpose of an honest benchmark. The likely explanation is resource pressure on the entry-tier instance under concurrent writes.

Surprise #1: Java Does Not Like Strict 256 MB Containers

One of the most interesting findings came from Docker Neo4j.

CognoDB’s free tier gives you:

• 0.5 vCPU
• 256 MB RAM
• 1 GB disk

I wanted the local Docker Neo4j instance to be as close to that as possible.

So I first capped the Docker container at 256 MB.

Neo4j crashed during startup.

It did not fail during the benchmark. It failed before it could serve queries.

The fix was to distinguish between two different memory concepts:

• container memory ceiling
• database JVM heap

The final Docker configuration used:

• 512 MB container ceiling
• 256 MB Neo4j JVM heap max
• 50 MB Neo4j page cache

That distinction mattered.

The database heap was still limited to 256 MB, but the JVM needed additional headroom just to run.

This became one of my favorite architectural observations from the project:

A database memory limit is not always the same thing as a process memory limit.

It also highlighted a difference between lightweight engines and JVM-based engines. CognoDB’s free tier operates within a very small total footprint, while Java-based Neo4j requires more baseline headroom.

Surprise #2: Free-Tier Latency Is Often About Fixed Overhead

The traversal results were interesting because latency barely changed as query complexity increased.

If the graph traversal itself were the dominant cost, we would expect three-hop queries to be noticeably slower than one-hop queries.

But for several platforms, they were almost identical.

That suggests that the measured latency was dominated by fixed overhead:

• network round-trip time
• request handling
• query planning
• serialization
• scheduling
• free-tier throttling

This is an important lesson for anyone reading database benchmarks.

A slow result does not always mean the engine is slow.

It may mean:

• the instance is tiny
• the network path is distant
• the CPU is burstable
• the environment is shared
• the platform is throttling free-tier traffic

Honest benchmarks need to say that clearly.

Surprise #3: Docker Is Useful, but It Is Not a Fair Cloud Comparison

Docker Neo4j had the lowest latency in almost every category.

But that result must be interpreted carefully.

Docker ran on the same machine as the benchmark client.

That means:

• no internet round-trip time
• no cloud routing
• no managed-service request overhead

Docker results are useful as a local engine baseline, but they are not directly comparable to cloud platforms over a network.

This is one of the reasons I included Docker in the first place.

It helped separate two questions:

1. How does the engine behave locally?
2. How does the managed platform behave over a real network path?

Those are related, but they are not the same.

The Importance of Percentiles

One of the requirements I’m glad the assignment enforced was percentile reporting.

Docker Neo4j had very low p50 latency:

• around 3 ms for lookups
• around 14 ms for aggregation

But its p95 latency jumped to around 80 ms in several workloads.

That gap matters.

Averages could make Docker look almost perfect. But p95 revealed occasional pauses, likely related to JVM behavior, caching, or local scheduling.

In production, users often notice the tail more than the average.

A system that usually responds in 5 ms but occasionally responds in 100 ms can feel worse than a system that consistently responds in 30 ms.

Percentiles expose that.

What the Results Suggest

Under the conditions of this benchmark:

Memgraph

Memgraph showed strong ingest throughput and strong cloud latency. Its in-memory architecture appears to benefit both batch loading and read workloads.

Neo4j Aura

Aura showed strong managed-cloud behavior, especially considering it was running on a free tier. It scaled well under concurrency and completed mixed workloads with zero recorded errors.

Docker Neo4j

Docker Neo4j produced the lowest local latency, but its results include a localhost advantage and should not be treated as a direct cloud comparison.

Neo4j Sandbox

Sandbox behaved like a managed trial environment with relatively high fixed request latency. It is useful for evaluation, but its internal resource guarantees are not publicly detailed.

CognoDB

CognoDB completed all benchmark workloads with zero mixed-workload errors. Its free tier is intentionally small, and under the tested network path it showed higher request latency and lower throughput than some other platforms.

The important caveat is that this reflects the free-tier conditions and client network path used in this test. It does not necessarily represent CognoDB’s behavior under larger instances, closer regions, or production configurations.

Honest Caveats

This benchmark has limitations, and they matter.

Free tiers are noisy

Free instances may be shared, throttled, or burstable. Results can vary over time.

Network path matters

Cloud platforms were tested from a residential internet connection. Different regions or networks could produce different results.

Not all specs are public

Some platforms do not expose exact vCPU, RAM, or storage details for free tiers.

Docker has a localhost advantage

Docker Neo4j avoids cloud network latency, so it should be interpreted as a local baseline.

Docker needed more container memory than CognoDB

Neo4j’s JVM required extra headroom beyond the 256 MB database heap.

Warm numbers only

The reported latency numbers are warm-run numbers. Cold-start behavior was not separately benchmarked.

One full measurement pass

Each read workload used 100 iterations, but the full suite was not repeated across multiple days due to the 48-hour assignment window.

Engine diversity

Aura, Sandbox, and Docker all use Neo4j-based engines. They were included to compare different deployment models. Memgraph provided a different in-memory architecture. Future work could include additional distinct engines such as FalkorDB, ArangoDB, NebulaGraph, or TigerGraph.

What I Would Improve Next

If I had more time, I would extend the benchmark in several ways.

Add more distinct engines

FalkorDB, ArangoDB, NebulaGraph, TigerGraph, and Kùzu would make the comparison more diverse.

Run repeated full-suite passes

Multiple runs across different times would help measure variance.

Add cold-start benchmarking

Cold-start latency is important for serverless and scale-to-zero environments.

Collect platform metrics

Where available, I would collect:

• CPU usage
• memory usage
• disk usage
• request errors
• saturation metrics

Generate charts automatically

The current report uses Markdown tables and simple text charts. A chart generator would make the results easier to read.

Test larger datasets

The current dataset fits free tiers. A larger dataset would stress indexing, memory, and storage more aggressively.

The Bigger Lesson

The biggest lesson from this project was not “Database X is faster.”

The bigger lesson was that benchmarking is a systems problem.

You are not only measuring the database engine.

You are measuring:

• the network
• the driver
• the query language
• the instance size
• the storage engine
• the runtime
• the cloud tier
• the client machine
• the workload design
• the honesty of the methodology

If you ignore those things, your benchmark may be technically reproducible but practically misleading.

If you document them, your benchmark becomes useful.

That was the spirit of this assignment.

Not to hide the messy parts.

Not to smooth over the caveats.

Not to declare a winner.

But to measure carefully, explain clearly, and leave the next engineer enough information to reproduce the work.

Repository

The benchmark harness, results, and full documentation are available here:

GitHub repository: https://github.com/kalbashi09/Benchmark

The README includes:

• full results tables
• platform specs
• dataset details
• query workloads
• methodology
• caveats
• reproducibility instructions

If you are evaluating graph databases, I hope this helps you ask better questions before trusting any benchmark.

Including mine.