Ask a model for five fields and you get back one string. Between that string and your database sits a parser, and a parser fails in two ways that are not versions of each other. A loud failure throws, or returns the wrong field count, and costs you a retry. A silent failure returns exactly k confident fields with one of them wrong, and costs you a row, forever.
So this page builds fourteen real encoder/decoder pairs — a real JSON.parse, an RFC 4180 scanner with quote-doubling, a tag scanner, a line scanner — and enumerates every producer state rather than sampling it, which is why a rate of 0.00% here means zero and not "none seen": https://dev48.infy.uk/prompt/day70-delimiter-design.html
The probe that needs no assumptions at all
One record, every value boring: not a comma, newline, pipe, quote or hash anywhere in it. The only variable is what the model wrote around the answer. Let it do the single most common thing a model does, which is not to hallucinate but to be polite:
Sure! Here are the fields you asked for:
5 of the 14 parsers hand back silently wrong data. Not an error — data. Every single-line format absorbs the greeting into field one and a sign-off into field k, so the count is right and the parse succeeds. Allow the other five ordinary wrappings and it is 8 of 14, with exactly 3 surviving all six — and every survivor is a format with an explicit closing token.
That reframes the standard defence of tags. Prose in front of the answer is harmless to ### customer headings and <customer> tags alike, so keying does work. Prose behind it separates them: a ### section has a start and no end, so every "let me know if you need anything else" lands inside your last field, with every key present, known and correctly counted. Keying buys immunity to what comes before your answer. Termination buys immunity to what comes after.
The free fixes beat the paid ones, three times
| change | delivered records | silent corruption | what it costs |
|---|---|---|---|
delete .filter(x => x !== '') from the newline parser |
34.48% → 64.63% | 3.83% → 0.00% | 0 tokens |
strip the fence before JSON.parse
|
85.42% → 92.85% first try | — | 108.2 → 99.6 tokens per correct record |
| escape the pipe, as the guides say | +2.80 points | +0.69 points | +26 instruction tokens per call |
| entity-escape the XML | +0.00 points | +0.00 | +22 instruction tokens, forever |
| two retries on comma-delimited output | +0.000 records | — | 73.28% of the token bill |
Thirty points and an entire safety column for removing eighteen characters from a parser. A free fix that makes the bill smaller. And a retry is a fresh draw over the model's behaviour, so it can fix a fence and it cannot fix a comma — the comma is in your customer's name, and 47.50% of five-field records here fail in every producer state, which makes their retry budget an exact zero rather than a small number.
What the measurement contradicted
I built the escape convention as the fix. Set the wrapping dial to zero and it delivers 99.54%: every collision in the data handled, completely, exactly as advertised. Turn the dial back to 8% and it delivers 91.58%, and the whole of the missing eight points is the greeting. It solves 100% of the problem it was designed for and 0% of the problem that actually fires.
It also adds 0.69 points of silent corruption, which I did not expect and which is the honest cost of the idea. An un-escaper is a new component with a failure the plain format never had: the model writes C:\temp raw, the un-escaper helpfully removes the backslash, and you store C:temp — five characters of plausible garbage where a hard error would have been better. Adding a decoder adds a way to be wrong.
564 in-page assertions and 536 in the verifier. One file, inline CSS, no external asset.
Part of a from-scratch series — one prompting technique a day, measured rather than described: https://dev48.infy.uk/promptfromzero.php
