We Ran Our Search Engine Backwards

Run the query rate cuts against our REST API: it finds "Fed cuts rates again."
Register the same query on a WebSocket subscription: the article sails right past.
Silence. Same product, same query syntax, same article, two different answers to
"does this match?"

We shipped that for a year.

Here's the engineering problem underneath. We ingest around ten thousand news articles
a day, in bursts, because news doesn't believe in uniform distribution. A few hundred
standing queries like rate cuts AND source:www.reuters.com are listening over
WebSockets and webhooks. Within seconds of an article entering the system, it has to
reach exactly the subscribers whose query matches it. No misses, no spam.

So: for every incoming article, how do you compute who gets notified?

It looks like a search problem, and it almost is, just mirrored. Search is one query
against many documents. This is one document against many queries.

A subscription is a search query that never stops running

Quick context so the rest makes sense. I run finlight, a
financial news API. Articles flow through an ingestion pipeline: scraped, enriched
with sentiment and company tags, then indexed into OpenSearch (the Elasticsearch
fork; everything here applies to both). Customers consume them in two ways:

• Pull: a REST API with full-text search over that OpenSearch index.
• Push: WebSocket streams and webhooks. You register a query once, in the same query syntax the REST API uses, and every future article that matches is pushed to you within seconds of ingestion.

A subscription, in other words, is just a search query that never stops running.
Because both modes share one query syntax, our docs have always given users a simple
tip: test your query against the REST API first, because what search returns is what
your socket will deliver. File that sentence away. It's going to come back with a
knife.

Somewhere between ingestion and delivery, then, a matching step has to answer: which
of the N registered queries does this one article satisfy? Everything that follows is
about that one step.

Wrong answer #1: query the search index for every article

The naive approach: we already run OpenSearch for our REST search. When an article
arrives, loop over all subscriptions and run each stored query against the article
index, then check if the new article is in the results.

At the time we were ingesting 4,000-5,000 articles per day. With N live subscriptions,
that's N × 4-5k full-text queries a day, not spread out nicely but clumped into spikes
whenever a news wave hits, hammering the same cluster that serves customer search
traffic.

You can refine it: filter every stored query down to the one new document's ID, so
each check touches almost nothing. That variant is a legitimate, battle-tested
pattern, and it would have limped along fine at our size. But it still runs N queries
against the live cluster for every article, all to answer a yes/no question about
one document you're already holding in your hand: does this document match this
query?

So no, not fatal. Wasteful. And wasteful was enough to send us looking for something
cheaper. The expensive lesson was waiting on the cheaper path.

Wrong answer #2: match in application code

So we did what almost every alerting feature in history does: we matched in application
code.

You've probably written this exact thing. Load the active subscriptions, and for every
incoming item, run a filter function:

function matches(article, sub) {
  const text = (article.title + ' ' + article.content).toLowerCase();
  return sub.keywords.every((k) => text.includes(k.toLowerCase())) &&
    (sub.sources.length === 0 || sub.sources.includes(article.source));
}

Keyword alerts in a Slack bot, price alerts in a shop, job alerts on a listings site:
under the hood it's nearly always this shape. toLowerCase(), includes(), maybe a
regex on an ambitious day. And it's genuinely attractive: microseconds per check,
zero extra infrastructure, zero load on your search cluster.

Our version was an industrialized flavor of the same idea. We already had a parser that
compiled our query syntax into OpenSearch DSL for the REST API, so we gave it a second
backend that emitted JavaScript predicates as strings, stored those in DynamoDB next to
each subscription, and revived them with new Function() per article. (Yes, eval-ing
strings from a database. They came from our own compiler, not from users, so injection
wasn't the issue. But it remained the kind of architecture you explain to another
engineer in a slightly lowered voice.)

The flavor doesn't matter, though. Hand-rolled includes() chain or compiled
predicate, it's the same category: matching in application code, next to a search
engine that matches differently. Ours ran in production for about a year.

The drift

Here's what we underestimated: not a bug, but a category difference.

When OpenSearch matches rate cuts, the text has been through our analyzer chain:
tokenized, lowercased, unicode-normalized, stemmed. "cuts", "cut" and
"cutting" collapse into the same root. Word order and punctuation are handled by query
semantics, not by luck.

An application-code matcher does none of that. Whatever the flavor, it's a string
check. Which means:

• REST API, query rate cuts → finds the article "Fed cuts rates again", the exact pair from the top of this post. Stemming maps cuts→cut, rates→rate, and multi-word queries compile to AND over stemmed terms; trivial match for a search engine.
• WebSocket, same query → the literal substring rate cuts appears nowhere in "Fed cuts rates again". Silence.

Then the support messages started:

"Why didn't I receive this article over my WebSocket? When I run the exact same query
against your REST API, it's right there."

This one hurt more than a normal bug report. Remember the tip from our docs, the one
about validating your query against the REST API first? Customers were doing exactly
that. The validation workflow we ourselves recommended was structurally incapable of
working.

We sat down and catalogued the differences between the two matchers, and found 13
distinct classes of divergence. A sample (a few of these depend on how we configured
our analyzers; not all of it is out-of-the-box behavior):

Row three is the support ticket. The other twelve were waiting their turn.

The five not shown: nested company queries, wildcards, stopwords, a filter bug,
scoring. We had already been patching these one at a time, each fix small, correct,
and doomed, before we sat down and made the full list. Thirteen individually
patchable divergences added up to one conclusion: we were rebuilding a search
engine's analysis chain in application code, badly, and it would never converge.
The drift wasn't a defect. It was the architecture.

And here's the uncomfortable generalization:

If your product has a search bar backed by a real engine and alerts matched by
includes(), you have this bug. Your users just haven't diffed the two channels
yet.

The breaking point: multilingual support

For English-only, you can almost live in denial. Then we decided to support more
languages, including Chinese, Japanese and Korean.

CJK languages don't put spaces between words. Tokenizing them isn't a regex problem;
it's dictionary- and model-driven segmentation. In OpenSearch you reach for dedicated
analyzers: kuromoji for Japanese, nori for Korean, smartcn for Chinese, plugins all of
them, and each a serious piece of engineering in its own right. And every language
brings a bag of quirks: compound-word splitting, reading forms, stopword curation,
width normalization. Japanese alone will humble you.

Now look at that from the predicate matcher's perspective. Our options:

1. Rebuild all of that in Node, inside a Lambda, and keep it bit-compatible with OpenSearch forever. A hobby project's worth of computational linguistics per language, with "stays identical to the real engine" as an eternal, manual QA problem.
2. Embed a different search library in the matcher. But a different engine means different analyzers and different semantics. We'd be institutionalizing the drift instead of eliminating it.

Spot the common flaw: both options mean tediously aligning a second matching engine
with OpenSearch, and never being done. Consistency across channels was the actual
product requirement, the one thing that support message demanded, and neither option
could deliver it. We genuinely evaluated alternative engines and walked away every
time with the same bad taste.

And then our prayers were answered

I wish I could say we designed our way out. The truth is humbler. While digging through
OpenSearch's documentation for something else entirely, we found a feature we had never
heard of:

The percolator.

We had spent weeks agonizing over how to keep two matching engines consistent, and
the tool we were already running in production could natively do the thing we
needed. Not a plugin. Not a sidecar. A built-in field type sitting in the docs,
shipping in core since 2016 with ancestors back to Elasticsearch 0.15 in 2011,
waiting for us to scroll far enough.

The pattern even has a name: prospective search, sometimes reverse search. It's
the pattern behind Google Alerts-style systems; Google even shipped a Prospective
Search API on App Engine once. Lucene has an entire library for it (Luwak, since
absorbed into Lucene as the Monitor module). We had simply never needed the words, so
we had never searched for them.

A normal index stores documents and you throw queries at it. A percolator index stores
queries, as documents in a dedicated percolator field type, and you throw a
document at it:

PUT /subscriptions
{
  "mappings": {
    "properties": {
      "query":  { "type": "percolator" },
      "title":  { "type": "text", "analyzer": "english" },
      "source": { "type": "keyword" }
    }
  }
}

PUT /subscriptions/_doc/sub_42
{
  "query": {
    "bool": {
      "must": [{
        "match": {
          "title": { "query": "rate cuts", "operator": "and" }
        }
      }],
      "filter": [
        { "term": { "source": "www.reuters.com" } }
      ]
    }
  }
}

GET /subscriptions/_search
{
  "size": 1000,
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "title": "Fed cuts rates again",
        "source": "www.reuters.com"
      }
    }
  }
}

Three details are load-bearing. The analyzer: the stemming that started this whole
story lives in that line. The operator: and: every term must match, which is exactly
when stemming matters, because without stemming rate never matches rates, and one
failed term sinks the whole query. And the size: the default is 10, and with enough
subscriptions a truncated hit list is silently missed notifications.

The response is the set of stored queries matching this document. Which is literally
the question from Wrong Answer #1, "does this document match this query?", asked N
times at once, natively, by the engine itself. The mirror image from the top of this
article isn't an analogy. It's the feature.

Why this killed the alignment problem dead

The mechanism matters: when you percolate a document, OpenSearch builds a tiny
in-memory index from it using the percolator index's own mapping and analyzers
(which mirror the article index's), then runs the stored queries against it. Same
tokenizers, same stemmers, same normalization as our article indices. Not "kept
compatible with" OpenSearch, but OpenSearch itself. There is no second engine to
align. What remains is config-level: each percolator index must keep mirroring its
article index's analyzers. A chore, but greppable and testable, unlike behavioral
drift.

Which means all the language machinery is defined once and inherited by both
directions: the kuromoji tokenization chain for Japanese, nori's compound-splitting
modes for Korean, smartcn plus a curated stopword list for Chinese, the stemmers and
unicode normalizers for everything else. Onboarding a new language means writing its
analyzer config once, and interactive search and real-time matching both speak it,
guaranteed to agree on what matches, from day one. Under the old architecture, every
piece of that machinery would have needed a hand-written, forever-maintained
JavaScript twin.

Because our REST API and our subscription registration already shared one query parser,
both directions now compile to the same DSL and execute on the same engine. The REST
validation workflow we recommend to users went from aspirationally true to
guaranteed by construction (with one scoring caveat, covered in the trade-offs
below).

Production: 10,000 articles a day, 5-20 ms a batch

The migration itself was pleasantly boring, the best kind. Today's shape:

• One percolator index per language, each mirroring the analyzer configuration of its article index. The subscription's language decides where its query is stored; the article's language decides which index it's percolated against.
• Subscriptions sync via change streams. Registering a WebSocket query or webhook writes to DynamoDB exactly as before; a small Lambda tails the table streams and upserts or deletes the corresponding percolator document. Subscription ID = document ID, so the sync is idempotent, and the realtime API's write path never touches OpenSearch directly. (The stream is eventually consistent; a fresh subscription starts matching within the sync lag, typically well under a second.)
• Batched matching. Articles leave the enrichment pipeline in small batches; a dispatcher Lambda percolates each batch in a single _msearch: one sub-request per article, routed to that article's language index. A typical percolate batch completes in 5-20 ms, and the whole matching workload sits well under one percent of cluster load.

Part of why it's so cheap is a lovely detail of the percolator itself: at registration
time the engine extracts terms from every stored query and uses them to pre-select
candidates, so only queries that could plausibly match ever run against the in-memory
document. Matching an article against hundreds of stored queries, a corpus totaling a
few megabytes, is nothing like querying millions of articles.

The new Function() matcher (the second parser backend, the predicate strings, the
matcher loop) was deleted. Nobody held a funeral.

Ingest volume has roughly doubled since the migration, to around 10,000 articles a
day, and the matching layer is the one part of the pipeline that has never needed a
second thought.

The recipe, if you're building this

Stripped of our war story, here's the general pattern for real-time alerts on any
OpenSearch/Elasticsearch-backed product:

1. Create a percolator index that mirrors your document index: same analyzers, same field mappings for everything queryable. Add a field of type percolator for the query itself, plus whatever metadata you need (subscription ID, channel, owner).
2. Sync subscriptions into it. On create/update/delete, upsert or remove one document: the compiled query plus metadata, with the subscription ID as _id so the sync is idempotent. Tail your subscription store's change stream (we use DynamoDB Streams) or write through an outbox; just don't couple your API's write path to the cluster.
3. Percolate every incoming document. Batch with _msearch; each response is the list of subscription IDs to notify. Set size well above your worst-case match count: the default is 10, and truncated hits are missed notifications. Hand the result to your delivery fan-out.
4. Keep non-search concerns out of the stored queries. Authorization, quotas, dedup: apply them in application code after matching, or every policy change becomes a reindex.
5. Going multilingual? One percolator index per language, each mirroring that language's analyzer config. Route by the subscription's language when storing, by the document's language when percolating.

That's the entire architecture.

The honest trade-offs

The percolator isn't magic, and you should hear the sharp edges from me:

• Matching is effectively binary. Percolation does score its matches, but against a single document there's no meaningful relevance cutoff. Our REST search drops low-scoring hits below a quality threshold; percolation has no equivalent. For alerting that's usually what you want, but it is a semantic difference between pull and push, accepted consciously.
• Mapping changes fan out. Every article field that should be queryable in subscriptions must be added to every language's percolator index. Adding two metadata fields once touched over twenty index definitions. The consistency guarantee has a maintenance tax.
• Language switches need care. A subscription lives in exactly one language index. Change a subscription's language and you must delete-then-register, or you leak an orphaned query that matches forever.
• Authorization stays outside. Which sources a customer may receive depends on their plan, not their query. Bake that into stored queries and you'll be reindexing customers every time pricing changes. Percolation stays about search semantics; access control runs in application code afterwards.

What I'd tell past-me

1. Don't rebuild your search engine's semantics in application code, not even with an innocent includes(). Analyzers, stemming, CJK segmentation: that's not "string matching with extra steps". Your reimplementation starts drifting the moment you ship it.
2. Consistency has to be a property of the architecture, not of your diligence. If two code paths must agree, make them the same code path. "We'll keep our clone aligned with the real engine" is a promise you have to re-keep after every single upgrade, and one day you won't.
3. Read your tools' feature lists, all the way down. We evaluated entire alternative engines while the answer sat in the docs of the database we already operated. Sometimes the answer to your prayers has been shipping since before your product existed.

Same parser, same DSL, same analyzers, same engine. Just run backwards. The support
messages stopped.

finlight.me is a financial news API with full-text search and real-time delivery over
WebSockets and webhooks. If you want to watch the percolator's output land on a socket,
the live demo is the fastest way.