Building an investing knowledge graph, part 4: building the resolver

Part 3 ended with a question I left open: how does the threshold actually get set, and what happens to the registry when new articles keep coming in? That's what this one covers.

The cost of getting it wrong in both directions

Before I touch thresholds, it's worth being specific about what failing in each direction costs.

A false merge combines two distinct entities into one node. In the Samsung SDI case from part 3, that would mean battery-supply disruptions show up when you traverse from the semiconductor business. The graph gives you an answer. The answer is wrong. You don't necessarily know it's wrong without already knowing the answer, which defeats the point.

A false split keeps the same entity as two separate nodes. "Samsung Electronics" and "the largest memory chipmaker in the world" stay disconnected. Edges pile up on each independently. A traversal from one doesn't reach the other. The graph gives you a partial answer. You lose coverage.

These aren't symmetric. A false merge injects noise that's hard to detect. A false split just means a disconnected node — you miss some connections, but you don't get fabricated ones.

So I set the threshold conservatively. Pairs below a high confidence mark stay split. Some real aliases end up never linking. That's the tradeoff I chose: lower recall, higher precision. For a graph I'm using to make judgments about which news matters, a confidently wrong edge is worse than a missing one.

What the endpoint does

The resolution service exposes a /v1/splink-pairs endpoint. It takes a list of candidate entity mention strings and returns a score for each pair, along with whether that pair crosses the registry threshold.

A minimal request looks like this:

POST /v1/splink-pairs
{
  "candidates": [
    "Samsung Electronics Co.",
    "the largest memory chipmaker in the world",
    "Samsung SDI",
    "TSMC"
  ]
}

The response returns a score matrix for pairs that the model considers worth evaluating — not every combination, just the ones above a blocking threshold that filters out obviously unrelated pairs before the full model runs.

{
  "pairs": [
    {
      "left": "Samsung Electronics Co.",
      "right": "the largest memory chipmaker in the world",
      "match_probability": 0.96,
      "decision": "match",
      "canonical_id": "company:samsung_electronics"
    },
    {
      "left": "Samsung Electronics Co.",
      "right": "Samsung SDI",
      "match_probability": 0.12,
      "decision": "split",
      "canonical_id": null
    }
  ]
}

The caller gets a decision (match or split) and the canonical entity ID when there's a match. If the match is against a known registry entry, the alias gets recorded. If it's a new pair the model hadn't seen before, that pair gets added to the evidence set for future model updates.

How the registry grows

When the investing knowledge graph pipeline processes a new article, it extracts entity mentions and runs them through the resolver. Three things can happen:

Known alias. The string matches an existing entry in the alias table. Lookup is fast, no model call needed. The mention gets written to the graph under the existing canonical ID.

Unknown mention, matches existing entity. The string isn't in the alias table, but the model scores it as a probable match against an existing entity. The alias gets added to the registry. Future articles that use the same string skip the model call.

Genuinely new entity. The model doesn't find a confident match against anything in the registry. A new canonical entry gets created. It starts small — one mention, no resolved aliases — and accumulates evidence as future articles mention the same company.

The registry currently has 47,853 resolved entities and 47,883 aliases. A lot of those started as single-mention nodes. Some have since merged as more articles confirmed the connection.

A few early decisions are probably still wrong. In the first batches I was more aggressive with merging, before I tightened the threshold. There are likely some nodes that should be split. I know this in general; I don't know specifically which ones without re-running the full corpus against the current model, which I haven't done.

What Splink is doing under the hood

Splink is the open source probabilistic record linkage library the model is built on. It uses DuckDB as the computation backend for the pair scoring, which handles the candidate generation and blocking step — filtering the space of possible pairs before running the full model.

For corporate entity mentions, the features that ended up mattering:

• String similarity on the canonical name: catches abbreviations and common shorthand
• Alias coverage: once a surface form is in the registry, future instances match by lookup rather than model score — this is how "the largest memory chipmaker in the world" eventually becomes reliable without its string similarity to "Samsung Electronics" ever improving
• Token overlap on co-occurring named entities within the same article: an article about DRAM yields tends to mention different companies than an article about battery chemistry
• Base rate weighting: a rare entity (one mention, no confirmed aliases) shouldn't get absorbed into a high-frequency entity just because they share a token

Splink learns the feature weights from labeled pairs. The training set was several hundred manually verified matches and non-matches from the first two thousand articles. Not large. But I had enough confirmed Samsung/Samsung SDI pairs to get the context features weighted correctly.

The point where this became a service

Somewhere around the third project where I had the same entity resolution problem — merging customer records from two systems, with the same fragmentation-versus-false-merge tradeoff — I stopped treating this as a per-project function and wrapped it into a standalone FastAPI service.

The knowledge graph pipeline now calls it over HTTP. Other things call it the same way. The registry is shared across callers. A match that one pipeline discovers helps every other caller that sees the same entity later.

That service is what I'm calling ER API. It runs on Railway, it's live, and the registry it's backing is the same one that's kept the investing knowledge graph from turning into a tangle of phantom nodes.

Next: what it takes to keep this running in production — the parts that only showed up after the first real caller wasn't me.

Built on Splink for probabilistic record linkage. Part 1 is here. Part 2 is here. Part 3 is here.