When an LLM wraps JSON inside XML tags, markdown fences, or SQL fragments, json-repair scores 25/25 on the wrapper-tagged cases in MALFORMED-300. jsonshim — our own normaliser — scores 17/25. The stdlib controls score 0/25.
This is one category out of twelve in a 300-case conformance corpus for malformed LLM JSON output. Every parser was run once, with the same grading contract, on the same hardware. The full breakdown is public: leaderboard.json.
What "wrappers" means here
Each of the 25 cases is valid JSON buried inside another format the model often emits:
- XML or HTML tags around the payload
- Markdown code fences with the wrong language tag
- SQL
SELECT ... AS jsonenvelopes - Log-line prefixes before the object
The recoverable answer is the inner JSON object. A parser that returns the wrapper verbatim, parses the wrapper as JSON, or throws without extracting the payload scores 0.
Full wrappers column (25 cases)
| parser | language | exact |
|---|---|---|
| json-repair | python | 25/25 |
| jsonc-parser | javascript | 18/25 |
| jsonshim (Toolkit Labs) | python | 17/25 |
| jsonrepair | javascript | 12/25 |
| pyjson5 | python | 8/25 |
| json5 | python | 8/25 |
| json5 | javascript | 8/25 |
| dirty-json | javascript | 8/25 |
| json-loose | javascript | 6/25 |
| simplejson | python | 4/25 |
| partial-json | javascript | 4/25 |
| hjson | python | 4/25 |
| best-effort-json-parser | javascript | 4/25 |
| untruncate-json | javascript | 0/25 |
| partial-json-parser | python | 0/25 |
| json.loads (stdlib) | python | 0/25 |
| hjson | javascript | 0/25 |
| dirtyjson | python | 0/25 |
| demjson3 | python | 0/25 |
| commentjson | python | 0/25 |
| JSON.parse (stdlib) | javascript | 0/25 |
json-repair wins this column because it strips common wrapper patterns before parsing. jsonshim was built for type coercion and truncated payloads, not wrapper extraction — the README names wrappers as a known weak category (17/25).
Overall score still favours jsonshim
Across all 300 cases, jsonshim leads at 282/300 (94.0%). json-repair is second at 265/300 (88.3%). The wrappers column is the main category where that order inverts.
Category-level pages with the same numbers: toolkitlabs.org/leaderboard/.
Try 12 cases free
The scorer, generator, and 12 open cases are CC0. All 300 run in the free kit with answers sealed.
python3 score.py --corpus open12.jsonl --parser jsonshim:loads
python3 score.py --corpus open12.jsonl --parser json
The full 300
The remaining 288 labelled cases with rationale for each label — €29 single developer, €99 team/CI licence. Instant download after payment.
MALFORMED-300 on Toolkit Labs · Buy — €29
The scorer, generator, and 12 open cases stay CC0 forever whether you buy or not.
