My tests could fail. They still could not tell me I was wrong.

A test proves your code does what you meant. It cannot prove that what you meant was correct.

I could have written that sentence a year ago. I still shipped on the wrong side of it this week.

Here is the assertion I was leaning on:

assert.equal(sumOf(splits), transaction.total)

A transaction split across several categories has to add up to the total. That is a real invariant, and the test catches plenty. What it cannot catch is the one thing I was actually unsure about, because both sides of that equation are produced by my code, using a sign convention I chose. Get the convention backwards and both sides flip together. The test stays green.

The two formats disagree about what a number means

I maintain qbofile, a browser based converter for the file formats small business accounting runs on. This particular conversion goes from IIF, which QuickBooks Desktop has used since the 1990s, into QIF, which Quicken and GnuCash read.

Spend $150 at one shop and split it between two categories. Here is IIF:

TRNS   ...   Amex Platinum      -150.00
SPL    ...   Office Supplies     100.00
SPL    ...   Meals                50.00

Double entry. The account side is negative, the category side is positive, and the whole block sums to zero.

QIF takes the same event from one point of view only, the account's:

T-150.00
SOffice Supplies
$-100.00
SMeals
$-50.00

The $ lines are negative here, and they sum to the T line rather than cancelling it. So the converter negates every split amount on the way across. One character:

L.push(`$${formatQifAmount(-s.amountCents)}`);

That character is a claim about how another company's file format works. I read the spec, I read other people's files, and I was fairly confident. Confident is not the same as measured.

The mutation test proved the wrong thing

Before trusting a test I like to break the code on purpose and check that the test notices. So I deleted the minus sign. Four tests went red, including the one above. Restore it, all green again.

That is a useful result. It rules out the test being vacuous, which is a real failure mode and more common than people admit.

But look at what it actually establishes. Flip a correct sign and the tests go red. Flip an incorrect sign and the tests also go red. The mutation test tells you the assertion is wired to the code. It is completely blind to whether the code is on the right side of the truth. I had proved my test could fail, and then quietly treated that as having proved my belief.

So I asked the software that has to read the output

The only authority on whether a QIF file is right is a program that imports QIF files. I built one file designed so that a sign error would be impossible to miss:

• one credit card account, so the file also exercises the !Type:CCard header
• two separate split transactions
• one category that receives money from both of them — $100 from one, $120 from the other

That last point is the whole design. It gives me a number that only comes out right if the sign is right, and it comes from adding across two transactions rather than reading one back:

Then I handed it to GnuCash. Import, no warnings, no "split does not balance". Office Supplies: 220.00. The account came in as a credit card rather than a bank account, so the type header was read too.

Fifteen minutes. The belief is now a measurement.

The bug I found on the way to the test

Deciding what was worth testing meant reading the whole chain again, and that is where the actual defect turned up. It had nothing to do with signs.

QIF has two separate fields: P for the payee, M for the memo. IIF carries both, in NAME and MEMO. The generator wrote one line:

L.push(`P${sanitizeText(tx.description)}`);

and upstream, description was defined as MEMO || NAME. So for any transaction that had a memo, the memo took the payee field and the actual payee was dropped on the floor.

before   PSUPPLIES RUN            <- memo posing as the payee, COSTCO gone
after    PCOSTCO / MSUPPLIES RUN

Nothing errors. The importing software has no way to know a field went missing, and neither does the person looking at their register afterwards. That is the failure mode this entire project exists to fight, and I had built one.

It was also the second time that day I had assumed a downstream generator could hold everything the upstream parser produces. The rule I wrote down: when you wire two components together, open the field lists at both ends and diff them. Do not just read the part in the middle that you are writing.

Which beliefs are worth fifteen minutes

You cannot run every assumption past real software; most of the time the unit test genuinely is the last word. What decided it here was a question I now ask explicitly:

Does this line rest on a belief about someone else's software?

Sign conventions in a foreign file format. Which header field another program actually reads. What an importer does with a value it does not recognise. Those are not properties of my code, so my code cannot testify about them, and a passing suite is the wrong kind of evidence entirely.

I had learned this once already and forgotten it. An earlier version of the same converter emitted an account type declaration, on the shared understanding that QuickBooks would create missing accounts with the type you declare. Every test passed. On a real copy of QuickBooks, every invented account came out as a bank account regardless of what the file said.

What makes this worth the interruption is the shape of the trade. Fifteen minutes buys an answer that no amount of additional test writing can produce, and the thing you are buying insurance against does not announce itself. A sign convention that is backwards does not crash anything. It produces a file that imports cleanly, into a register that looks plausible, for someone who has no way to check.

The IIF file I built for the GnuCash test, and the four things I checked after importing it, are in the repo notes. If you have a QIF importer that disagrees with any of this, I would like to hear about it.