You pasted the file
and asked why the totals are wrong.
The answer came back confident
and slightly beside the point.
You went round three times,
sharpened the question,
got better prose
and the same wrongness.
The file was two thousand lines.
About nine hundred of them arrived.
Nothing told you that.
This is the quiet failure
in every tool that reads on your behalf.
A retriever returns three passages
because three was the number
in a config file
somebody set in June.
A file reader stops at a line limit
that was chosen with logs in mind.
A window drops the oldest turns
to make room for the newest,
including the one
where you stated the constraint
that the whole question rests on.
An indexer skips anything
over a size,
and the large file
is the interesting file.
None of them raise an error.
They all report success.
And the model does
the only thing it can,
which is answer from what it holds,
with no way of knowing
that the part that mattered
fell off the back.
To it, an absence
is indistinguishable
from a thing that is not there.
So you are not arguing
with a bad answer.
You are arguing about
a different program
from the one you meant to show it.
Verify the input like input.
Ask it to quote the last line
of what it received,
and compare that with the file.
Ask what the final function
is called.
Count the characters you sent,
if that is the only number
you can get hold of,
and check it against the source.
When a tool fetches for you,
read what it fetched,
not the conclusion
built on top of it.
And prefer the smaller ask.
One function, entire,
beats a whole service
trimmed by a rule
you have never read.
The habit is older
than any of this technology.
Before you debate the output,
prove the input arrived.
Otherwise you will spend
an afternoon doubting
your own reasoning,
and the machine will turn out
to have been perfectly correct
about a program
that does not exist.
– Serguey Asael Shinder
