I built a 300-case labelled corpus of Unicode strings that break naive text handling — combining marks, ZWJ sequences, astral codepoints, bidi controls, zero-width characters, homoglyphs, NFKC folds, case mapping traps, and exotic whitespace. Twenty cases are free (CC0). The rest are EUR 19 on toolkitlabs.org.
This is not a Unicode conformance suite. ICU and the UCD own that job. This corpus answers a narrower production question: what does your len(), truncate, fold, and normalize code actually do on strings a user can paste?
Reproduce the free slice in three commands
curl -sLO https://toolkitlabs.org/unicode300/open20.jsonl?s=devto-4479156
curl -sLO https://toolkitlabs.org/unicode300/check.py
curl -sLO https://toolkitlabs.org/unicode300/controls.py
python3 check.py --fn controls:ascii_fold --cases open20.jsonl
open20.jsonl is 14,387 bytes — two cases per category, the lowest ids in each. check.py runs your function over every case and counts three failure modes: raised, returned non-string, or returned a string that disagrees with the labelled trap on the properties the case is meant to test.
What the three shipped controls do on all 300
I ran the published controls against the full corpus (not just the open 20). Numbers below are from the site's published table, one run, nothing tuned after:
| control | failures on 300 |
|---|---|
controls:ascii_fold |
247 |
controls:codepoint_len |
198 |
controls:nfc_then_len |
156 |
The point is not that any control is "good." The point is that different naive handlers fail on different subsets, and you cannot infer coverage from one happy-path emoji test.
A case that breaks len() alone
From open20.jsonl, case u300-0001:
{"id":"u300-0001","string":"cafe\u0301","trap":"len() counts codepoints, not what a reader sees..."}
Five codepoints, four NFC codepoints, six UTF-8 bytes. Truncating to four codepoints can leave a combining acute standing alone.
What EUR 19 buys (and what stays free forever)
-
Free forever:
open20.jsonl,check.py,controls.py, scorer docs — CC0, no account. -
EUR 19 one-time: the other 280 labelled cases as JSONL, flat CSV with measured properties, and corpus SHA-256
e3e1c76e836bd259c571967f0210bc7cd1e081b808720b137ae0cafed934b780over the 300-line JSONL exactly as shipped.
UNICODE-300 on Toolkit Labs · Buy — EUR 19
Everything on the page reproduces without the paid file: the free zip, the open slice, and the checker are enough to score your own function before you buy anything.
