Around pair 400 of the manual annotation batch, I got to a Korean holding company and its Japanese parent. The Japanese database had the company listed as a joint venture, the Korean filing described it as a wholly-owned subsidiary. Same entity, but the ownership description was genuinely different between sources. I marked it as a match since the names clearly referred to the same company, but I wrote a note that I still don't know what to do with.
That was about a month in. I ended up reviewing around 2,000 pairs by hand before I had enough coverage to feel reasonably confident in the KR-JP and KR-ZH models.
The reason it got to manual annotation: I couldn't find a public labeled dataset for Korean-Japanese-Chinese cross-lingual corporate name matching. For English-only ER, there are open sources. For this specific domain, I didn't find anything usable.
The Korean-English piece came together faster than I expected. Korean financial disclosure filings (DART) require companies to report both the Korean legal name and the English name. That turned out to be a clean source of verified KR-EN pairs. The company certified both names in a regulatory document, so I didn't have to guess whether they referred to the same entity.
def fetch_dart_name_pairs(company_code: str) -> dict:
url = "https://opendart.fss.or.kr/api/company.json"
params = {"crtfc_key": DART_API_KEY, "corp_code": company_code}
resp = requests.get(url, params=params)
data = resp.json()
return {
"ko": data.get("corp_name"),
"en": data.get("corp_name_eng"),
}
Scraping recent DART filings got me a few thousand KR-EN positive pairs. Combined with negatives generated by sampling companies from different industry sectors, the KR-EN Splink model got decent coverage early.
The KR-JP and KR-ZH pairs didn't have an equivalent source. I went looking for companies with cross-listed securities that would appear in databases across multiple languages. Found maybe 800-900 verified triplets that way. Not many, but enough to get an initial model to a usable state for the core coverage.
The 2,000 manual annotations were the gap-filler. Mostly companies in the near-threshold band of early model runs, which sounds principled but in practice meant sitting with a lot of cases where I wasn't sure. Japanese holding company naming conventions have edge cases I'd never thought about before. A company could be listed under a regional holding name in Japanese filings but appear under the global brand in Korean corporate registries. Not wrong, just a different layer of the corporate structure.
The KR-EN model is the one I trust most. The KR-JP and KR-ZH models have held up for the core coverage but I've seen them struggle on cases that weren't well represented in the training set. I add pairs when I find errors in production, but it's slow.
I build er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.
