Enterprise Knowledge Base (04): HyperGraphRAG Benchmark — Multi-hop Reasoning with Hypergraph Structure

What This Article Is About

This is the fourth benchmark in the series. The previous three articles tested four frameworks:

• QAnything v2 (vector RAG, article 02)
• LightRAG 1.5.6 (graph + vector hybrid, article 02)
• GraphRAG 3.1.1 (LLM-driven knowledge graph, article 03)
• HippoRAG 2.0 (hippocampus-inspired PPR graph diffusion, article 03)

This one tests the last framework: HyperGraphRAG 1.0.6 (NeurIPS 2025).

What HyperGraphRAG Actually Does

Traditional knowledge graph edges are binary: entity A → entity B. This means a relationship like "A, B, and C jointly participate in an event" must be decomposed into multiple binary edges — and the decomposition loses the joint semantic context.

HyperGraphRAG uses hyperedges to address this: a single hyperedge can connect any number of entities at once.

Traditional graph:  A → B, B → C, A → C  (3 separate edges, context fragmented)
Hypergraph:         {A, B, C} connected by a single hyperedge (joint context preserved)

The pipeline from documents to hypergraph:

Documents → LLM extracts entity sets (co-occurring entities per passage, not pairwise)
          → Build hypergraph (each entity set = one hyperedge)
          → BGE vectorizes entities
          → Query: vector recall for seed entities → hypergraph diffusion
                 → extract relevant passages → LLM answer

The theoretical expectation: for "multi-entity joint participation" relationships, hypergraph extraction should surface more signal than binary graph extraction.

Deployment

Installation

HyperGraphRAG is not on PyPI. There is no pip install hypergraphrag, and pip install -e . fails because there is no setup.py or pyproject.toml. The working approach: add the source directory to PYTHONPATH and install the requirements from the README:

pip install -r requirements.txt   # graspologic, nano-vectordb, etc.

# Run with:
PYTHONPATH=/path/to/HyperGraphRAG python run_eval.py

SiliconFlow API: Two Hidden Constraints

Constraint 1: Content filter blocks environment variable patterns (HTTP 400, code 20015)

The test documents contain many .env example snippets like:

EMBEDDING_MODEL=jina-embeddings-v4
EMBEDDING_BINDING=jina
EMBEDDING_ASYMMETRIC=true

The ALLCAPS=value format triggers SiliconFlow's content safety filter. Fix: regex-strip any lines matching sensitive patterns before sending embedding requests.

Constraint 2: bge-large-en-v1.5 has a ~512-token input limit

Testing revealed that inputs over roughly 1000 characters return 400. Dense markdown tables (every | and % consumes a token) can exceed 512 tokens at just 900 characters.

Final setting: truncate each text to 700 characters (word-boundary truncation), giving a buffer for high-density content.

Network Stability During Long Runs

GLM-4-flash and SiliconFlow both produced intermittent ConnectTimeout and HTTP 500 errors during the 7-hour indexing run. Two necessary changes:

1. Set LLM client timeout to 180s, with 6 retry attempts (10/20/30/40/50/60s backoff)
2. Switch from one-shot rag.insert(all_docs) to per-document insert with a progress file: each successfully indexed document is immediately checkpointed; a crash and restart continues from where it left off

Without both changes, finishing a 443-minute index build in a single run is unrealistic.

Benchmark Results

Numbers

Answer matching uses Jaccard keyword overlap, not LLM judge; LLM is GLM-4-flash throughout

Multi-hop: 0.171 — Tied with LightRAG, Not the Breakout

The theoretical advantage of hyperedges didn't materialize here. HyperGraphRAG's 0.171 on multi-hop is within noise of LightRAG's 0.178.

A few plausible explanations:

• The test documents are technical reference materials (configuration guides, API docs). Multi-entity joint relationships are rare in this genre — that's precisely where hyperedges help most, but the signal isn't there to exploit
• Hyperedge quality depends on LLM extraction quality; GLM-4-flash's performance limits what the graph can represent
• BGE truncation to 700 characters may have dropped some semantic signal from longer entity descriptions

Index Build Time: 443 Minutes

GraphRAG took 31 minutes, HippoRAG took 36 minutes. HyperGraphRAG took 443 minutes — roughly 7.4 hours.

The cause: HyperGraphRAG issues multiple LLM calls per chunk to extract entity sets, and llm_model_max_async=1 (GLM rate limit). With 124 chunks, ~4 LLM calls each, plus the accumulated timeout retries, the total inflates significantly.

In a production setup with a higher-concurrency LLM (GPT-4o), the indexing time would likely drop to 30–60 minutes. GLM's low concurrency is the amplifying factor here.

Boundary Refusal: 26.3% — Better Than Expected for Graph RAG

5 of 19 boundary questions (26.3%) were correctly declined. This is higher than GraphRAG's 15.8% and HippoRAG's 5.3%.

Likely reason: HyperGraphRAG's hybrid mode (local + global search) may be more willing to return "not found in the knowledge base" when neither local graph traversal nor global context returns relevant hits.

Full Five-Framework Comparison

Combining all four articles:

Patterns across the five frameworks:

1. Index cost and multi-hop performance are decoupled: GraphRAG achieved the highest multi-hop score (0.211) with a 31-minute build. HyperGraphRAG spent 443 minutes and got 0.171. More indexing time does not buy better retrieval.

2. LightRAG has the best cost-effectiveness: 8-minute build, second-best multi-hop (0.178), lowest P90 latency (19.4s). Its only weakness is low refusal rate (10.5%) — the graph tends to produce an answer regardless of relevance.

3. Refusal capability is a design decision, not an emergent property: QAnything's 26.3% refusal rate comes from explicit confidence-threshold filtering. Graph RAG frameworks don't ship with a refusal mechanism and show near-zero rates as a result.

4. Single-hop: graph structure adds nothing: HippoRAG's 0.122 is the best among graph RAG approaches, but vector RAG and graph RAG both cluster in the 0.08–0.12 range. For direct factual retrieval, knowledge graph overhead doesn't help.

Which Framework Should You Choose?

Based on the five-framework data:

Default to LightRAG for most use cases: fast indexing, strong multi-hop, lowest latency. Add a confidence post-processing step to improve refusal behavior if needed.

Choose GraphRAG if:

• Multi-hop reasoning is your primary use case (policy cross-referencing, causal chain analysis)
• You have LLM budget for the indexing cost
• You're using a model with reliable structured output (GPT-4o, Claude) to enable community reports; the GLM structured output issues in this test disabled that feature entirely
• 30-minute build times are acceptable

Choose QAnything if:

• You need strong out-of-the-box refusal capability with minimal configuration
• You want a self-contained deployment without managing graph infrastructure

Wait on HyperGraphRAG: the hyperedge concept is theoretically compelling, but version 1.0.6 has real production friction:

• No PyPI package; source-only installation
• 7-hour build time in this configuration
• Multi-hop performance didn't surpass LightRAG in a technical document domain
• Check back when a production-grade release lands

Code Highlights

Full code at llm-in-action/kb-04-hypergraphrag-eval/.

Text truncation + sanitization

import re

_SENSITIVE_LINE_RE = re.compile(
    r'[^\n]*(?:'
    r'api[_\-]?key|secret|token|password|Authorization'
    r'|sk-[A-Za-z0-9]{10,}'
    r'|[A-Z][A-Z0-9_]{3,}=[^\s\n]+'  # ENV_VAR=value
    r')[^\n]*',
    re.IGNORECASE,
)
_MAX_CHARS = 700  # bge-large-en-v1.5 via SiliconFlow ~512 tokens

def _truncate(text: str) -> str:
    text = _SENSITIVE_LINE_RE.sub("[REDACTED]", text)
    if len(text) <= _MAX_CHARS:
        return text
    truncated = text[:_MAX_CHARS]
    last_space = truncated.rfind(" ")
    return truncated[:last_space] if last_space > 0 else truncated

Per-document insert with checkpoint resume

def build_index(rag, doc_files: list[Path]) -> float:
    progress = load_progress()  # load completed list
    for fpath in doc_files:
        if fpath.name in progress["indexed"]:
            continue  # skip already done
        content = fpath.read_text().strip()
        rag.insert([content])              # insert one at a time
        progress["indexed"].append(fpath.name)
        save_progress(progress)            # checkpoint immediately

Next: Series Wrap-Up — benchmark methodology limitations, what this test set measures (and doesn't), and how to design a production-grade RAG evaluation from scratch.

Check out PrimeSkills — a curated marketplace of AI agents and skills validated in real-world, enterprise-grade workflows. Not demos — things that actually work in production.

Find more on my Homepage