Vector Databases Are Not Magic, Here's What's Actually Happening Under the Hood

You've seen the tutorials. Spin up Pinecone, call .upsert(), do a similarity search, ship it. Everyone claps. The demo works.

Then you take it to production and it starts lying to you.

Results that look semantically relevant but aren't. Queries that should match something and return nothing. Latency that makes your users think the app crashed. And the worst part - you don't know why, because the vector database feels like a black box with a fancy API.

This article is about opening that box.

What a Vector Database Actually Is

Let's be honest about what "vector database" means, because the term is doing a lot of marketing work right now.

At its core, a vector database is an index optimized for approximate nearest neighbor (ANN) search over high-dimensional float arrays. That's it. The "database" part - persistence, CRUD, filtering, transactions - is infrastructure wrapped around that core capability.

When you store an embedding, you're storing a point in N-dimensional space (typically 768, 1536, or 3072 dimensions depending on your model). When you query, you're asking: "which stored points are closest to this query point, by some distance metric?"

The challenge? Doing exact nearest neighbor search at scale is O(N * D) - linear in your corpus size times the dimensionality. For a million 1536-dim vectors, that's ~6 billion float comparisons per query. At millisecond latency requirements, that's a hard no.

ANN algorithms trade a small amount of accuracy for massive speed gains. Understanding this trade-off is the first thing most tutorials skip - and it's where production bugs hide.

The Index Is the Product

The algorithm your vector DB uses to build its index determines everything: speed, recall, memory usage, and how it degrades under pressure.

HNSW (Hierarchical Navigable Small World)

This is what most modern vector DBs use by default (Qdrant, Weaviate, Milvus, pgvector with the right extension). HNSW builds a multi-layer graph where:

• The top layer is sparse - only a few highly-connected "hub" nodes
• Each lower layer gets progressively denser
• Querying starts at the top and greedily navigates down toward the nearest neighbor

Think of it like a highway system. You jump on the highway (top layer), drive toward your destination, exit at the right interchange, and then use local streets (bottom layer) for precision.

Key parameters you need to know:

# Qdrant example
from qdrant_client.models import VectorParams, Distance

client.create_collection(
    collection_name="my_docs",
    vectors_config=VectorParams(
        size=1536,
        distance=Distance.COSINE,
        hnsw_config={
            "m": 16,          # Number of edges per node. Higher = better recall, more memory
            "ef_construct": 100,  # Construction-time beam width. Higher = better index quality, slower build
        }
    )
)

# At query time
results = client.search(
    collection_name="my_docs",
    query_vector=query_embedding,
    limit=10,
    search_params={"ef": 128}  # Runtime beam width. Higher = better recall, slower query
)

m and ef_construct are set at build time and can't change without rebuilding your index. If you're seeing poor recall in production and you set m=4 to save memory, that's your culprit.

IVF (Inverted File Index)

Used by FAISS and as an option in pgvector. Divides the vector space into Voronoi cells (clusters), assigns vectors to their nearest centroid, then searches only a subset of cells at query time.

# FAISS IVF example
import faiss
import numpy as np

dimension = 1536
n_clusters = 1024  # Number of Voronoi cells

quantizer = faiss.IndexFlatL2(dimension)
index = faiss.IndexIVFFlat(quantizer, dimension, n_clusters)

# Must train before adding vectors
index.train(training_vectors)  # Needs representative data
index.add(corpus_vectors)

# nprobe = how many cells to search. More = better recall, slower
index.nprobe = 32
distances, indices = index.search(query_vector, k=10)

IVF gotcha: the cluster centroids are learned during training. If your data distribution shifts significantly (new document types, different topics), your centroid structure becomes suboptimal and recall tanks. You don't get an error. You just quietly get worse results.

Distance Metrics: You're Probably Using the Wrong One

Most people use cosine similarity because the tutorial said so. Here's when that's wrong.

OpenAI's text-embedding-3-* embeddings are normalized to unit length. Cosine similarity on unit vectors is mathematically equivalent to dot product. Using cosine adds a normalization step that's pure overhead.

# If you're using OpenAI embeddings, use dot product
# In Qdrant:
VectorParams(size=1536, distance=Distance.DOT)

# In pgvector:
# Use <=> for cosine, <#> for negative inner product (dot), <-> for L2
SELECT content, embedding <#> query_embedding AS score
FROM documents
ORDER BY score
LIMIT 10;

The difference in latency is small at low scale. At 10M+ vectors, it's measurable.

The Recall Problem Nobody Talks About

Here's a thing that will haunt you: your ANN search does not always return the true nearest neighbors.

It returns approximate nearest neighbors. That's the A in ANN. By definition, you may miss results that should have ranked in your top-K.

How bad is it? It depends on your index config and your data. You can measure it:

import numpy as np
from qdrant_client import QdrantClient

def measure_recall(client, collection_name, test_queries, ground_truth_ids, k=10):
    """
    Compare ANN results against brute-force exact search.
    ground_truth_ids: list of lists, true top-k ids per query
    """
    hits = 0
    total = len(test_queries) * k

    for query, true_ids in zip(test_queries, ground_truth_ids):
        ann_results = client.search(
            collection_name=collection_name,
            query_vector=query,
            limit=k
        )
        ann_ids = {r.id for r in ann_results}
        hits += len(ann_ids & set(true_ids))

    return hits / total  # recall@k


# A well-tuned index should hit 0.95+ recall@10
# If you're at 0.85 or below, tune ef or m

Production target: ≥ 0.95 recall@10. Anything below that and your RAG pipeline is silently missing relevant context before GPT-4 ever sees it.

Hybrid Search: The Architecture You Should Actually Be Using

Pure vector search has a well-known failure mode: it doesn't handle rare terms well.

If your corpus contains "RFC 7807 Problem Details" or a specific error code like E_INVALIDARG_0x80070057, embedding similarity will dilute the match across semantically adjacent concepts. A user querying for the exact string gets mushy results.

The solution is hybrid search: combine dense vector search with sparse BM25-style keyword search, then fuse the rankings.

from qdrant_client import QdrantClient
from qdrant_client.models import (
    SparseVectorParams, VectorParams,
    SparseIndexParams, Distance, NamedVector, NamedSparseVector
)

# Qdrant supports both dense and sparse vectors natively
client.create_collection(
    collection_name="hybrid_docs",
    vectors_config={
        "dense": VectorParams(size=1536, distance=Distance.COSINE),
    },
    sparse_vectors_config={
        "sparse": SparseVectorParams(index=SparseIndexParams(on_disk=False))
    }
)

# At insert time, generate both representations
from fastembed import SparseTextEmbedding, TextEmbedding

dense_model = TextEmbedding("BAAI/bge-small-en-v1.5")
sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")

text = "RFC 7807 Problem Details for HTTP APIs"
dense_vec = list(dense_model.embed([text]))[0]
sparse_vec = list(sparse_model.embed([text]))[0]

# At query time, use Reciprocal Rank Fusion (RRF)
from qdrant_client.models import Prefetch, FusionQuery, Fusion

results = client.query_points(
    collection_name="hybrid_docs",
    prefetch=[
        Prefetch(query=dense_vec.tolist(), using="dense", limit=20),
        Prefetch(
            query=SparseVector(indices=sparse_vec.indices.tolist(),
                               values=sparse_vec.values.tolist()),
            using="sparse", limit=20
        ),
    ],
    query=FusionQuery(fusion=Fusion.RRF),
    limit=10
)

RRF (Reciprocal Rank Fusion) combines the rank lists without needing score normalization. The formula is simple:

RRF_score(d) = Σ 1 / (k + rank_i(d))

Where k is a constant (usually 60) and rank_i(d) is the document's rank in each result list. Documents appearing in both lists get a significant boost.

Hybrid search consistently outperforms pure dense search on real-world corpora by 5–15% on NDCG@10 - especially for domain-specific or technical content.

Metadata Filtering: The Performance Trap

Vector DBs let you pre-filter by metadata before (or after) the ANN search. This sounds simple. It's actually one of the most common performance footguns.

Pre-filtering (filter before ANN): Apply your metadata filter first, reduce the candidate set, then run ANN on the smaller set.

Problem: if your filter is very selective (e.g., user_id = "abc123" in a multi-tenant system), the candidate set might be tiny. HNSW graph navigation assumes a large, connected graph. A sparse subgraph destroys recall.

Post-filtering (ANN then filter): Run ANN on the full corpus, retrieve top-N, then apply filter. You need to over-fetch significantly to compensate for filtered-out results.

# Qdrant handles this with "indexed" payload fields
# Always index fields you filter on
client.create_payload_index(
    collection_name="my_docs",
    field_name="tenant_id",
    field_schema="keyword"  # or "integer", "float", "geo"
)

# Qdrant uses a smart filtering strategy:
# If filter is selective → brute force on filtered set
# If filter is broad → HNSW with post-filter
# It decides automatically based on cardinality estimates

results = client.search(
    collection_name="my_docs",
    query_vector=query_embedding,
    query_filter=Filter(
        must=[FieldCondition(key="tenant_id", match=MatchValue(value="abc123"))]
    ),
    limit=10
)

Rule of thumb: if your filter reduces the corpus below ~1000 vectors, you're effectively doing brute-force search. That's fine - just know it and set expectations accordingly.

The Chunking Strategy You Need to Revisit

This isn't vector DB internals, but it's so deeply related that skipping it would be malpractice.

Your retrieval quality is bounded by your chunking quality. The vector DB can only return what you gave it.

Most tutorials show:

# The naïve approach that everyone copies
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = text_splitter.split_text(document)

The problems:

• Fixed-size chunks break semantic units arbitrarily
• A sentence spanning a chunk boundary gets split into two orphaned halves
• 500 tokens might be too large for precise retrieval, too small for necessary context

Better: semantic chunking

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95  # Split when semantic shift exceeds 95th percentile
)

chunks = splitter.split_text(document)

This embeds sentences, calculates cosine distance between adjacent sentence pairs, and splits at significant semantic shifts.

Even better: store both chunk and parent document

# "Small-to-big" or "Parent Document Retrieval"
# Store small chunks for precise matching
# But return the parent document (or larger window) as context

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore

child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=InMemoryStore(),
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

Small chunks match with high precision. The returned context is the larger parent - so your LLM gets enough surrounding information to reason correctly.

Observability: What You Should Be Logging

If you're not measuring this stuff, you're flying blind:

import time
from dataclasses import dataclass
from typing import Optional

@dataclass
class RetrievalTrace:
    query: str
    query_embedding_ms: float
    search_ms: float
    num_results: int
    top_score: float
    bottom_score: float
    score_spread: float  # top - bottom; low spread = retrieval is uncertain
    filter_applied: Optional[dict]
    collection_name: str

def traced_search(client, collection_name, query_text, embed_fn, k=5, filter=None):
    t0 = time.perf_counter()
    embedding = embed_fn(query_text)
    embed_ms = (time.perf_counter() - t0) * 1000

    t1 = time.perf_counter()
    results = client.search(
        collection_name=collection_name,
        query_vector=embedding,
        limit=k,
        query_filter=filter
    )
    search_ms = (time.perf_counter() - t1) * 1000

    scores = [r.score for r in results]
    trace = RetrievalTrace(
        query=query_text,
        query_embedding_ms=embed_ms,
        search_ms=search_ms,
        num_results=len(results),
        top_score=scores[0] if scores else 0,
        bottom_score=scores[-1] if scores else 0,
        score_spread=(scores[0] - scores[-1]) if len(scores) > 1 else 0,
        filter_applied=filter,
        collection_name=collection_name
    )

    # Ship to your observability stack (Datadog, Langfuse, custom)
    log_trace(trace)
    return results

What to watch:

• score_spread near 0 means all results look equally similar - the query probably didn't match anything well
• top_score below your threshold (tune per model, but ~0.75 for cosine is a reasonable starting floor) means you're returning noise
• Embedding latency spikes often precede throttling errors from your embedding provider

The Stack Decision

Quick opinionated guide for 2026:

Don't use a vector DB for everything. If your corpus is under ~10,000 documents, cosine search over an in-memory numpy array with np.dot is fast enough and eliminates an entire infrastructure dependency.

import numpy as np

corpus_embeddings = np.load("embeddings.npy")  # shape: (N, 1536)
query_embedding = np.array(embed(query))        # shape: (1536,)

# Cosine similarity (assuming normalized vectors)
scores = corpus_embeddings @ query_embedding
top_k_indices = np.argsort(scores)[::-1][:10]

No database. No network calls. No ops burden. Just math.

What This Means for Your RAG Pipeline

Pull all of this together and you get a mental model for diagnosing RAG failures:

1. LLM gives wrong answer despite having the right docs? → Generation problem, not retrieval
2. Right docs never appear in retrieved context? → Check recall, check chunking, check distance metric
3. Results feel semantically correct but factually off? → Your chunks are too large; precision is suffering
4. Exact terms missing from results? → You need hybrid search
5. Multi-tenant data leaking across users? → Your metadata filter is wrong or not indexed
6. Works in dev, breaks in prod? → Data distribution shift. Retrain/rebuild index or tune ef/nprobe

Vector databases are not magic retrieval oracles. They're approximate spatial indexes with a product wrapper. Once you understand the approximation, the trade-offs, and the failure modes - you can actually build reliable systems with them.

If this was useful, I write about Python backend and AI engineering on dev.to. The good stuff is in the details.