Disclosure: I'm the founder of Face2social, a face-recognition search engine. This post is about the vector-search engineering behind it — no product pitch, just the stuff that kept me up at night (including the part where I tell you what we're bad at). It's also the write-up I wish I'd found when I started.
TL;DR
- A face becomes a vector. Similarity search becomes nearest-neighbor search in high-dimensional space.
- Brute force dies somewhere around a few million vectors. We're at ~1 billion.
- FAISS is what makes "compare against a billion" feel like "compare against a thousand."
- The real trick isn't one magic index — it's the trade-off triangle: speed ↔ memory ↔ recall. You don't get all three.
- And the biggest trade-off isn't even in the index. It's what you choose to put in it — more on that, honestly, near the end.
If you've ever wondered how "search by image" actually works at scale, this one's for you. 👇
The problem, honestly stated
Someone uploads a photo. We need to answer one question fast: does this face appear anywhere in a very large set of public profile pictures?
The naive version is easy to describe and impossible to ship:
Turn the uploaded face into a vector. Turn every known face into a vector. Compare the new vector against all of them. Return the closest matches.
That last step is the whole ballgame. Compare against a thousand faces? Trivial. A million? Now you're sweating. A billion? A single brute-force scan would have a user staring at a spinner long enough to close the tab, make coffee, and reconsider their life choices.
So the interesting engineering isn't "how do we compare two faces." It's "how do we not compare against all billion and still find the right one."
Step 1: a face is just a point in space
Before any search happens, every face — the uploaded one and every one we already know about — goes through the same pipeline:
detect the face → align it → turn it into a fixed-length vector (an "embedding").
The important intuition: two photos of the same person land close together in this space, even across different angles, lighting, crops, or filters. Two different people land far apart. "How similar are these faces?" collapses into "how close are these two points?" — a plain distance measurement.
That's the quiet superpower of embeddings: they turn a messy perceptual problem ("do these look like the same person?") into clean geometry ("what's the distance between these vectors?"). Everything downstream is just... finding nearby points. A lot of nearby points. Very fast.
(I'm deliberately staying vendor-neutral on the embedding model itself — pick your favorite; the search problem below is the same regardless.)
Step 2: why brute force falls off a cliff
Exact nearest-neighbor search means measuring the distance from your query to every stored vector, then sorting. It's O(n) per query, and each comparison is real work in a high-dimensional space.
Here's the mental model for why it doesn't scale:
- A thousand vectors: instant. Do it in a loop. Nobody cares.
- A million: still fine on a good box, especially batched.
- A hundred million: now every query is chewing through serious memory bandwidth, and your p99 latency is a horror story.
- A billion: the vectors alone don't comfortably fit in RAM on one machine, and a full scan per query is simply off the table for anything interactive.
The wall isn't CPU cleverness. It's physics — memory and bandwidth. You cannot touch a billion things per request and stay under a second. So you stop being exact.
Step 3: approximate is the whole point
This is the mindset shift that makes billion-scale search possible:
You don't need the mathematically perfect nearest neighbor. You need a very good neighbor, almost always, fast.
That's Approximate Nearest Neighbor (ANN) search, and it's where FAISS (Facebook AI Similarity Search) earns its keep. FAISS is a library built for exactly one job: nearest-neighbor search over massive vector sets, with a menu of indexes that each pick a different spot on the trade-off triangle.
Two ideas do most of the heavy lifting. Neither is exotic once you see the intuition:
1. Don't search everything — search a neighborhood.
Instead of scanning all billion vectors, you pre-cluster them into buckets during an offline training step. At query time you figure out which few buckets your query probably lives in, and you only scan those. You've quietly turned "compare against a billion" into "compare against a few million" — the same answer, a tiny fraction of the work. The knob for how many buckets you probe is your dial between speed and recall.
2. Don't store fat vectors — store compressed sketches.
Full-precision embeddings are heavy, and a billion of them is a lot of RAM. So you compress each vector into a compact code that's cheap to store and cheap to compare — trading a little precision for a massive memory win. This is what makes "a billion vectors" go from rent-a-data-center to actually-runs. You lose a hair of accuracy; you gain the ability to exist.
Stack those two — search a neighborhood, over compressed sketches — and a billion-vector search starts to feel routine. I'm intentionally not naming the exact FAISS index recipe we run in production (that part's ours to keep), but everything above is the standard playbook, and FAISS gives you all the building blocks out of the box.
Step 4: the trade-off triangle nobody escapes
Here's the thing I most wish someone had tattooed on my monitor early on. Every knob you turn is a negotiation between three things, and you only get to optimize two:
- ⚡ Speed — how fast a single query comes back.
- 🧠 Memory — how much RAM the index eats (this is your infra bill).
- 🎯 Recall — how often the true match is actually in your results.
Want blazing speed and tiny memory? Recall suffers — you'll miss real matches. Want near-perfect recall and speed? Get your wallet out for RAM. Want cheap and accurate? It'll be slow.
There is no free lunch. The entire job is choosing which corner to sacrifice for your use case — and for a face-search product, quietly missing a real match is the worst failure mode, so we bias toward recall and pay for it elsewhere. Your product's priorities will land you somewhere different, and that's the point: the "right" index is the one that fits your failure modes, not a leaderboard.
Step 5: one machine ends, sharding begins
Even compressed, a billion vectors plus the overhead to serve them will eventually blow past a single machine. So you shard: split the index into pieces across several machines, run the query against all shards in parallel, then merge the top results.
Conceptually it's the classic scatter–gather:
query ──► [ shard 1 ]─┐
├──► [ shard 2 ]─┤
├──► [ shard 3 ]─┼──► merge top-K ──► results
└──► [ shard N ]─┘
Your effective latency becomes the slowest shard plus a cheap merge — not the sum. Add shards to grow capacity, keep each shard small enough to stay fast, and you've got a search tier that scales sideways instead of falling over.
Step 6: the trade-off that actually defines the product
Everything above is about the index. But the decision that shapes your product most isn't how you search — it's what you put in the corpus. And this is where I should be straight with you, because it's our biggest limitation.
We index social media profile pictures. That's it. Instagram, Facebook, TikTok, X.
The upside is real: a narrow, dense, well-understood corpus means the index is tuned for one kind of photo, matches come back fast, and when someone asks "does this face have a social profile?" we're very good at answering it.
The downside is just as real, and I'd rather say it out loud than have a user discover it: we're blind to everything else. News articles, personal blogs, forums, company team pages, image boards, the open web in general — we don't see any of it. A face that's all over the web but has no social presence will come back empty from us, and that's not a bug we're about to fix; it's a direct consequence of the corpus we chose. Tools that crawl the open web will beat us on exactly those queries. They'll also index a much noisier space and inherit different problems, but that's their trade-off, not a defense of ours.
So: if your question is "is this person on social media, and where?" — that's our lane and we're strong in it. If your question is "where does this face appear anywhere online?" — we are the wrong tool, and you should use something that crawls the whole web.
I'm spelling this out because the corpus decision gets glossed over in most write-ups about vector search, and it's the one your users actually feel. A perfectly tuned index over the wrong data is still the wrong answer, delivered quickly. Recall against your index means nothing if the thing you're looking for was never in it. Define your corpus honestly, then tell people where its edges are.
What I'd tell past me
- The embedding model gets the glory; the index does the work. A great model that you can't search at scale is a demo, not a product.
- "Approximate" is not a compromise you apologize for — it's the enabling idea. Chasing exact nearest-neighbor at a billion scale is how you burn a quarter and ship nothing.
- Pick your sacrifice on purpose. Speed, memory, recall — decide before you tune which one you're willing to bleed, based on what "wrong" costs your users.
- Your corpus is a product decision, not a data decision. Narrow and deep beats broad and shallow — as long as you're upfront about the edges.
- FAISS is genuinely the boring-in-a-good-way choice. Battle-tested, flexible, and it lets you move the trade-off dials instead of reinventing them.
We run this in production at Face2social (US-only, for the privacy-law reasons you'd expect) to match an uploaded photo against public social profiles — mostly for catfish and romance-scam checks, and for people who want to know whether their own photos are being used on fake accounts. But honestly, swap "faces" for "product images," "audio fingerprints," or "document embeddings" and the entire playbook above is identical. Vector search at scale is one of those skills that quietly transfers everywhere.
If you're building anything that searches by similarity, learn FAISS, internalize the trade-off triangle, and be honest about your corpus. Future-you will thank present-you. 🙏
What are you using for large-scale vector search — FAISS, or something else? And how did you scope your corpus? Curious what trade-offs you landed on. Drop it in the comments.
