Whoosh on the no-GIL Python: pure-Python search that can finally use your cores

For fifteen years, the standard answer to "why is my pure-Python search slow to
index?" was: the GIL. Tokenizing, stemming, and building postings is all
CPU-bound Python, and only one thread gets to run Python at a time. Threads
didn't help. If you wanted parallelism you dropped into a C extension or shelled
out to a separate process.

Python 3.14 (stable since October 2025) ships an officially supported
free-threaded build — the no-GIL build from [PEP 703]. On it, pure-Python
CPU work can actually run on multiple cores at once. That is unusually good news
for a pure-Python full-text search library like
Whoosh (pip install whoosh3,
the maintained fork), because indexing is exactly the kind of per-token Python
work that used to be stuck behind the GIL.

But "no GIL" is not a magic @parallel decorator. You still have to structure
the work so threads don't stomp on each other. This post is the concrete
pattern — the one shipped as a worked example in the repo — for indexing a
corpus across threads safely, plus the honest performance caveats.

The one rule that makes it safe: one writer per thread

Whoosh's concurrency contract is small enough to memorize:

The writer is the sharp edge. A plain IndexWriter takes the index's write
lock and is not meant to be touched by two threads at once. So the trick is
don't share a writer. Give every worker thread its own sub-index in its
own directory, then merge the finished sub-indexes at the end. No shared
writer, no lock contention — the parallelism lives entirely in the fan-out.

Fan-out / fan-in

corpus  ->  split into N shards
        ->  N worker threads, each builds its OWN sub-index   (parallel, CPU-bound)
        ->  main thread merges the sub-indexes with add_reader()
        ->  one final, ordinary Whoosh index

The merge step uses writer.add_reader() — the same primitive Whoosh's own
multiprocessing writer uses to stitch segments together. It takes a read-only
reader from each shard, so again, no writer is ever shared.

The code

A worker builds one shard into its own directory. Only that thread ever touches
that writer, so the single-writer contract holds automatically:

from whoosh import index
from whoosh.analysis import StemmingAnalyzer
from whoosh.fields import ID, TEXT, Schema

def make_schema():
    # StemmingAnalyzer gives each thread genuine per-token CPU work —
    # exactly what free-threaded builds let you parallelize.
    return Schema(id=ID(stored=True, unique=True),
                  body=TEXT(analyzer=StemmingAnalyzer()))

def build_shard(schema, shard, subdir):
    ix = index.create_in(subdir, schema)   # this thread's own index
    w = ix.writer(limitmb=128)
    for doc_id, body in shard:
        w.add_document(id=doc_id, body=body)
    w.commit()
    ix.close()
    return subdir

Fan out with a thread pool, then fan in with add_reader:

from concurrent.futures import ThreadPoolExecutor

def build_parallel(docs, workdir, workers):
    schema = make_schema()                      # immutable -> shared safely
    shards = [docs[i::workers] for i in range(workers)]
    subdirs = [f"{workdir}/shard-{i}" for i in range(workers)]

    with ThreadPoolExecutor(max_workers=workers) as pool:
        built = [f.result() for f in (
            pool.submit(build_shard, schema, shards[i], subdirs[i])
            for i in range(workers))]

    # Merge: one read-only reader per shard, no writer shared.
    final = index.create_in(f"{workdir}/final", schema)
    w = final.writer(limitmb=256)
    for subdir in built:
        sub = index.open_dir(subdir)
        with sub.reader() as r:
            w.add_reader(r)
        sub.close()
    w.commit(optimize=True)
    final.close()

That's the whole pattern. The result is an ordinary Whoosh index — you search it
exactly as you would a serially-built one.

Correctness first: it's the same index either way

Parallelism that returns different results isn't a feature, it's a bug. The
merged index must be identical in behavior to a serially-built one — same doc
count, same query results. The shipped example pins that as a test:

def verify_equivalent(docs, final_dir):
    ix = index.open_dir(final_dir)
    with ix.searcher() as s:
        assert s.doc_count() == len(docs)
        # every query returns the same hits a serial build would
    ix.close()

The library's test suite runs this across even and uneven shard splits, so the
fan-out/fan-in never silently drops or duplicates a document.

Now the honest part about speed

Run the example
and it prints your build's GIL status via sys._is_gil_enabled(), times a
serial baseline against the parallel path, and tells you what to expect:

• On a normal (GIL) build, the parallel path is not faster — often a touch slower, because you pay the merge overhead without getting real parallelism. The GIL still serializes the per-token Python work. The example says so out loud rather than pretending otherwise.
• On a free-threaded build (3.13t/3.14t, GIL disabled), that same pure-Python indexing work is free to spread across cores. That is where the fan-out pulls ahead — and crucially, without a C extension releasing the GIL for you. The exact speedup depends on your cores, corpus, and analyzer chain, so measure it on your own hardware with the built-in harness rather than trusting a headline number.

The point isn't a magic multiplier. It's that a pure-Python library now has a
real, safe path to multi-core indexing — something that simply wasn't possible
while the GIL was mandatory.

Why this matters for pure Python specifically

Free-threading is a bigger deal for pure-Python libraries than for ones already
built on C extensions. A C-heavy library could always release the GIL around its
hot loop; a pure-Python one couldn't. So the no-GIL builds close a gap that hit
pure-Python code hardest. For a search engine you can pip install with zero
compiled dependencies, "indexing scales across cores" moves from impossible to a
documented pattern.

Whoosh's CI already runs the core test suite on the free-threaded 3.14t build
(gating) and 3.15t (early-warning) under a parallel test runner, specifically
to catch data races the GIL used to hide. The concurrency contract above is
documented in full,
and the parallel-indexing example ships in the repo so you can run it yourself.

Whoosh is a pure-Python full-text search library — no server, no compiled
extensions, no external services. I maintain the whoosh3 fork on PyPI. If the
free-threading direction is interesting to you, the
repo is where the work happens —
stars, issues, and PRs all welcome.