Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

A vector database can return 20 documents in 15 milliseconds and still give your RAG system a terrible answer.

The hard part of retrieval is rarely generating vectors. It is deciding whether the vectors are putting the right documents near the query.

That makes embedding evaluation an information-retrieval problem.

Once you see it that way, three metrics become particularly useful:

• Recall@k: did we retrieve the relevant information at all?
• MRR: how quickly did we find the first relevant result?
• NDCG: did we put the most useful results near the top?

Then there is a second problem.

What happens when your users search in Hindi for English documentation? Or type a product name in Latin script inside a Hindi sentence? Or use internal terminology that never appeared in the embedding model's training data?

And finally: when should you fine-tune the embedding model instead of changing something else?

Let's build the answer from the ground up.

1. An embedding model is really a ranking system

It is tempting to think of an embedding as a semantic fingerprint.

You encode:

"How do I rotate an API key?"

and get a vector such as:

[0.18, -0.07, 0.42, ...]

You encode every document chunk in your corpus, store those vectors, and retrieve the closest ones.

Usually the similarity is cosine similarity:

cos(q, d) = (q . d) / (||q|| * ||d||)

When vectors are normalized to unit length, cosine similarity becomes equivalent to the dot product:

cos(q, d) = q . d

So the system is doing something conceptually simple:

query
  |
  v
embedding
  |
  v
rank documents by similarity
  |
  v
top-k documents

The important word is rank.

Your application does not really care whether document A has a cosine similarity of 0.83 and document B has 0.79.

It cares whether A should appear before B.

This distinction has been present in information retrieval for decades.

In 2002, Kalervo Järvelin and Jaana Kekäläinen published work on evaluating retrieval systems using graded relevance: a document can be irrelevant, somewhat useful, highly useful, and so on. Their motivation was practical: retrieval systems produce too many results, so evaluation has to reward systems that put highly relevant material near the top.

Modern embedding search has the same problem.

The vector database is merely the mechanism.

Your real system is:

semantic query -> ranking -> useful context

That means you need a test set.

2. Build a retrieval test set before touching the model

Suppose you have a documentation system with 100,000 chunks.

You collect 500 real user queries:

"how to rotate api keys"
"why does websocket authentication fail"
"can I run the agent without public ports"
"delete an organization"
...

For each query, you identify the documents that actually answer it.

For example:

Query:
"how to rotate api keys"

Relevant:
doc_1842
doc_7119
doc_9320

That gives you a small retrieval benchmark.

You can now compare embedding models using the exact same queries.

This is much more useful than saying:

Model A "looks more semantic."

A useful benchmark should contain the ugly cases.

Include short queries.

Include vague queries.

Include terminology that only your users understand.

Include queries with multiple correct documents.

Include queries where the correct answer is buried inside a large document.

Include typos.

Include multilingual queries.

And keep a separate test set that you do not use while tuning the model.

A simple evaluation dataset might look like:

query_id: 17
query: "rotate api key"
relevant_docs:
  - doc_1842
  - doc_7119

query_id: 18
query: "websocket auth failure"
relevant_docs:
  - doc_921

Now your embedding problem becomes measurable.

3. Recall@k answers the first question: "Did we find it?"

Suppose there are five relevant documents for a query:

Relevant = {A, B, C, D, E}

Your embedding search returns:

Top 5 = [X, A, Y, Z, B]

You found 2 of the 5 relevant documents.

So:

Recall@5 = 2 / 5 = 0.40

In general:

Recall@k =
    relevant documents retrieved in top-k
    -------------------------------------
    total relevant documents

The intuition is:

Recall@k measures how much of the answer space you managed to bring into view.

Consider a RAG system retrieving 10 chunks.

If the answer requires information from one specific chunk and that chunk never appears in the top 10, your LLM cannot recover it.

A stronger language model does not magically fix missing context.

Why recall@k matters for RAG

Imagine 1,000 evaluation queries, with an average of 4 relevant chunks each.

That is approximately:

1,000 * 4 = 4,000 relevant chunks

If Recall@20 is 0.80, your retriever brought back roughly:

4,000 * 0.80 = 3,200

relevant chunks.

About 800 relevant chunks were missed.

That is a concrete failure budget.

Recall@k has an important limitation

It treats all relevant documents equally.

Suppose the relevant set is:

[A, B, C]

and two systems return:

System 1: [A, X, Y, Z, B]
System 2: [X, A, B, Y, Z]

At k=5, both have identical recall.

Yet System 1 puts the strongest result first.

Recall tells you whether useful material entered the candidate set.

It says much less about ordering.

That is where MRR and NDCG enter.

4. MRR asks: "How quickly did we find something useful?"

MRR stands for Mean Reciprocal Rank.

For one query:

reciprocal rank = 1 / rank_of_first_relevant_result

So if the first relevant result appears at:

rank 1 -> 1.00
rank 2 -> 0.50
rank 3 -> 0.33
rank 10 -> 0.10

For multiple queries, take the average.

Example:

Query 1 -> first relevant at rank 1 -> 1.00
Query 2 -> first relevant at rank 2 -> 0.50
Query 3 -> first relevant at rank 4 -> 0.25
Query 4 -> no relevant result      -> 0.00

Then:

MRR = (1.00 + 0.50 + 0.25 + 0.00) / 4
    = 0.4375

MRR is especially useful when your query usually has one obvious answer.

Think of questions such as:

"what is the default timeout?"
"where is the config file?"
"how do I reset my password?"
"what command starts the server?"

For these, getting the correct result at rank 1 is much better than getting it at rank 15.

But MRR ignores everything after the first relevant document.

Consider:

System A: [A, X, X, X, X]
System B: [A, B, C, D, E]

If A is relevant, both have:

RR = 1.0

MRR sees them as identical.

That is clearly wrong for many RAG applications.

If your answer depends on several pieces of evidence, you care about the whole ranking.

This is where NDCG becomes useful.

5. NDCG asks the more realistic question: "Did we rank the useful stuff properly?"

NDCG stands for Normalized Discounted Cumulative Gain.

The useful idea is simpler than the name.

You assign each retrieved result a relevance grade.

For example:

0 = irrelevant
1 = somewhat useful
2 = useful
3 = directly answers the question

Now imagine your ranking is:

rank 1 -> relevance 3
rank 2 -> relevance 2
rank 3 -> relevance 0
rank 4 -> relevance 1
rank 5 -> relevance 0

NDCG gives more credit to highly relevant documents near the top.

The underlying DCG calculation is:

DCG@k =
    sum from i=1 to k of
    (2^rel_i - 1) / log2(i + 1)

The pieces have intuitive meanings.

2^rel_i - 1

makes relevance 3 much more valuable than relevance 1.

And:

log2(i + 1)

discounts lower-ranked results.

So relevance at rank 1 counts more than relevance at rank 10.

Then normalize against the ideal ordering:

NDCG@k = DCG@k / ideal_DCG@k

Therefore:

NDCG@k = 1.0

means your results are ordered exactly like the ideal ranking.

A concrete example

Suppose the ideal relevance grades are:

[3, 2, 2, 1, 0]

Your system returns:

[1, 0, 3, 2, 0]

The system still retrieved good documents.

Recall might look decent.

MRR might also look decent because a relevant document appears early.

But NDCG drops because the grade-3 answer was pushed to rank 3.

That is often exactly what you want to punish in a RAG pipeline.

The metrics are telling you different things

Think of them as three different questions:

Recall@k -> Did we retrieve enough of the answer?

MRR      -> Did we find an answer quickly?

NDCG     -> Did we order useful answers correctly?

A good retrieval benchmark often uses all three.

A simple Python implementation makes the distinction concrete:

import math


def recall_at_k(ranked_ids, relevant_ids, k):
    relevant_ids = set(relevant_ids)
    retrieved = set(ranked_ids[:k])
    return len(retrieved & relevant_ids) / len(relevant_ids)


def reciprocal_rank(ranked_ids, relevant_ids):
    relevant_ids = set(relevant_ids)

    for rank, doc_id in enumerate(ranked_ids, start=1):
        if doc_id in relevant_ids:
            return 1.0 / rank

    return 0.0


def ndcg_at_k(relevances, k):
    def dcg(values):
        return sum(
            (2 ** rel - 1) / math.log2(i + 2)
            for i, rel in enumerate(values[:k])
        )

    actual = dcg(relevances)
    ideal = dcg(sorted(relevances, reverse=True))

    return actual / ideal if ideal else 0.0

Notice something important:

NDCG requires graded judgments.

If every document is simply relevant or irrelevant, NDCG still works, but you are throwing away useful information.

For many production systems, a practical judgment scheme is:

0 = does not answer the query
1 = related but insufficient
2 = useful evidence
3 = directly answers the query

That is often enough to make ranking failures visible.

6. Multilingual retrieval changes the problem

Now suppose your documentation is multilingual.

A user asks:

"How do I reset my password?"

The relevant document is written in Hindi:

"पासवर्ड भूल जाने पर आप अपना पासवर्ड रीसेट कर सकते हैं..."

A genuinely multilingual embedding model should place those texts near each other in vector space.

This is a harder problem than ordinary monolingual similarity because the model has to learn a language-independent semantic representation.

This became a major research direction before today's LLM era.

Mikel Artetxe and Holger Schwenk, for example, developed LASER, a multilingual sentence-embedding system covering 93 languages. Their work explicitly evaluated cross-lingual similarity and showed that a shared embedding space could support semantic search across languages.

But "multilingual" does not mean "equally good in every language pair."

You should test that yourself.

For example, your evaluation matrix might look like:

These numbers are illustrative, but the structure of the test is important.

The aggregate score could hide a serious problem.

Suppose:

English-English: 0.92
English-Hindi:   0.61
Hindi-English:   0.58

and 90% of your benchmark queries are English-English.

Your overall recall could still look excellent.

Your Hindi users would experience something very different.

What tends to break multilingual retrieval?

There are several recurring cases:

Different scripts

English: "insurance claim"
Hindi:    "बीमा दावा"

Transliteration

"बीमा"
"beema"
"bima"

Code-switching

"UPI ka transaction fail ho raha hai"

Named entities

A company, API, medicine, place, or product name may appear unchanged across languages.

Domain vocabulary

Your users may mix natural language with terms such as:

OAuth
JWT
webhook
protobuf
nginx

These are neither cleanly English nor cleanly Hindi.

For multilingual retrieval, evaluate by language pair and query type rather than relying on one global score.

The broader lesson came through later benchmark work too. The MTEB benchmark evaluated embedding models across many tasks and languages and found that no single embedding method dominated every task. Embedding quality is task-dependent.

That observation matters enormously when somebody says:

"We use the best embedding model."

Best for what?

7. When should you fine-tune the embedding model?

Fine-tuning is attractive because it feels like the direct solution:

retrieval is bad
        |
        v
fine-tune embedding model
        |
        v
retrieval gets better

Sometimes that is exactly right.

Often it is premature.

The first question should be:

What kind of retrieval error are you trying to fix?

Consider this failure:

Query:
"How do I rotate an API key?"

Returned:
"API authentication concepts"
"API key configuration"
"Creating API credentials"

The system understands the general topic but misses the specific operation.

This could be an embedding-model problem.

Now consider another failure:

Query:
"How do I rotate an API key?"

Relevant chunk exists.

The embedding retriever finds the correct document.

But your chunk is 3,000 tokens long and the relevant sentence is buried near the bottom.

Fine-tuning the embedding model will not solve the real problem.

You have a chunking problem.

Or perhaps:

Query:
"delete organization"

The system retrieves documentation for both:

delete organization
delete organization member

The semantic representation may be perfectly reasonable.

You may need metadata filters, query rewriting, or a reranker.

A useful decision sequence

Start with the cheapest intervention.

1. Check chunking
2. Check metadata filters
3. Check query preprocessing
4. Try a stronger embedding model
5. Add a reranker
6. Fine-tune the embedding model

The exact order can change, but the principle is:

Fix the smallest component that explains the failure.

Fine-tuning becomes attractive when errors are systematic

Suppose you have 5,000 real query-document judgments.

You repeatedly see:

"policy renewal"

being ranked below:

"policy purchase"

even though your users consider renewal documents clearly relevant.

That tells you something useful.

Your production relevance function differs from the semantic relationships learned by the base embedding model.

Now fine-tuning has a concrete target.

You can train on examples such as:

query:
"policy renewal"

positive:
"Renewing an existing insurance policy"

hard negative:
"Purchasing a new insurance policy"

The hard negative matters.

Random negatives are usually too easy.

If the model already knows that:

"dog food"

is unrelated to:

"database replication"

training on that pair teaches it almost nothing.

You want confusing examples:

positive:
"reset password for an existing account"

hard negative:
"change account email address"

The model needs to learn the boundary.

Fine-tuning has an economic cost

Suppose you want 2,000 evaluation queries.

You inspect the top 20 retrieved chunks for each query.

That is:

2,000 * 20 = 40,000 judgments

At just 10 seconds per judgment:

40,000 * 10 seconds = 400,000 seconds
                   ~= 111 hours

The expensive part of embedding work is often not GPU training.

It is producing trustworthy relevance data.

That means your benchmark itself is an asset.

Once you have it, you can evaluate:

Embedding A
Embedding B
Embedding C
Embedding C + reranker
Fine-tuned C
Fine-tuned C + reranker

using the same dataset.

There is also a production cost

Suppose you store 1 million chunks with 1,536-dimensional float32 embeddings.

Raw vector storage is approximately:

1,000,000 * 1,536 * 4 bytes
= 6.144 GB

before vector-index overhead, metadata, replication, and backups.

A 768-dimensional representation cuts the raw vector storage roughly in half.

This is why embedding evaluation is not purely about "which model has the highest score?"

You are optimizing a system with several variables:

retrieval quality
latency
embedding generation cost
storage
index size
reranking cost
engineering complexity

Suppose a new model improves Recall@20 from:

0.84 -> 0.87

but doubles embedding generation cost and increases query latency.

That improvement might still be worthwhile.

Or it might be irrelevant if a cheap reranker gets you:

0.84 -> 0.91

at acceptable latency.

You need measurements rather than intuition.

8. The practical evaluation loop

A production embedding evaluation system does not need to be elaborate.

Start with perhaps 300-1,000 real queries.

For each query, store:

query
language
relevant document IDs
optional relevance grades

Then run every candidate retriever over the same benchmark.

Track at least:

Recall@5
Recall@10
Recall@20
MRR
NDCG@10

For multilingual systems, break them down by language pair.

For domain-heavy systems, break them down by query category.

For example:

                Recall@10   MRR    NDCG@10
--------------------------------------------
Authentication    0.91      0.88     0.86
Billing           0.87      0.79     0.81
Deployment        0.83      0.76     0.77
Multilingual      0.62      0.51     0.55

Now you have something far more actionable than:

"The embeddings seem pretty good."

You can also investigate individual failures.

For every bad query, log:

query
top-k documents
similarity scores
relevance labels
language
chunk metadata

Then ask:

Was the answer absent from top-k?
Was the right document present but badly ranked?
Was the chunk itself poor?
Was the query ambiguous?
Was this a language-specific failure?

Those questions lead to different engineering fixes.

That is the main mindset shift.

Embedding evaluation is not a model leaderboard exercise.

It is debugging a ranking system.

The embedding model is one component inside that system.

And once you have a real benchmark, the fine-tuning question becomes much easier.

You can make the change, rerun the exact same queries, and ask:

Did Recall@20 improve?

Did NDCG improve?

Did multilingual retrieval improve?

Did we make another language worse?

Did latency or cost change?

Did the improvement survive on the held-out test set?

That is much more reliable than choosing an embedding model because its benchmark score looks good.

Conclusion

Embeddings turn semantic similarity into geometry.

Retrieval turns that geometry into a ranking.

Evaluation tells you whether that ranking actually serves your users.

Recall@k tells you whether the useful information entered the candidate set.

MRR tells you how quickly the first useful result appears.

NDCG tells you whether the most useful results were placed where users can actually benefit from them.

Multilingual evaluation tells you whether a single aggregate number is hiding failures across languages.

And a good benchmark tells you when fine-tuning is justified rather than merely tempting.

The most useful question is therefore not:

"Which embedding model should I use?"

It is:

"What retrieval behavior do my users actually need, and can I measure whether my system produces it?"

What retrieval metric or failure mode has mattered most in the RAG systems you have built?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

     

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.

blast-radius-demo.mp4

LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box	Visualize blast radius at a glance	Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)

Here's the goal:

• A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
• A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase: