An Agent That Counts My Receipts, Not My Claims

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

My ledger says three outside engineers found the same defect. Ask the agent how many locatable receipts support that row and it says one.

Finding B8 has 1 locatable comment (3ee98 by Pushpendra). Although three outside finders
are recorded, the receipts for Vinh Nguyen and quashudev were not found in the comment trees
searched.

It does not turn three recorded names into three verified receipts. The dataset stores finders and
receipts as different things, and the agent is not permitted to add them together.

It does that over a public Sanity record of 120 documents: 74 articles, 14 findings, 10 people,
3 patches, and 19 claims. The claim documents separately store asOf, status and
expiryStatus, rather than asking the model to reconstruct those distinctions from prose.

Three things it will tell you about its own author:

• the fix for finding B1 is not merged into origin/main — it reports inMain: false
• of 14 findings, 12 were raised by the 8 outside engineers and 2 are internal (one raised by my coding agent, recorded as Ka'el, and one by an internal audit). Two are implemented; twelve are not
• it cannot see what happened to that branch after the record was built, and says so

Demo

https://ask-the-record.vercel.app — no login, no user-supplied key, five buttons.

Measured answers on this deployment ran 13.8–33.8 seconds across the five questions — the raw
timings are in evidence/web-latency-2026-09-21.json. Evidence-bearing
answers must survive a retrieval check: a substantive verdict returned with no successful read is
marked a contract violation. INSUFFICIENT_EVIDENCE is allowed to abstain without pretending a
source was retrieved. Retrieval or auth failures return an error instead of an answer. Every
answer that renders cleanly is required to carry five fields:

ANSWER          the claim, stated plainly
SOURCES         what it read, and any source URL carried in the record
EVIDENCE DATE   the asOf recorded in the document — never today's date
VERDICT         STANDING · RETRACTED · SUPERSEDED · UNBUILT · EXPIRED
                NO_EXPIRY_SET · INSUFFICIENT_EVIDENCE
UNCERTAINTY     what this answer cannot establish. Never empty.

Press the Kubernetes button and it returns INSUFFICIENT_EVIDENCE and no number, because the
Knowledge Base evidence available to that question does not establish a count. It is not claiming
the dataset could never yield one. On the graded run, the agent abstained rather than supplying a
number; the validator does not independently establish that every abstention contains no unsupported claim.

The page also checks its own output. If an answer is missing a required field, has empty
uncertainty, or fails the citation requirement for the instrument that produced it — a URL for
dataset answers, an entry path and the Knowledge Base id for Knowledge Base answers — the page
shows the violation instead of rendering the answer cleanly. It checks citation presence and
syntax. It does not resolve URLs, and it does not yet prove that every emitted citation belongs
to the evidence actually retrieved.

Keys live server-side. The browser never receives one.

Code

https://github.com/keniel13-ui/ask-the-record

studio/     the schema — person · article · finding · patch · claim
harness/    a CLI version of the same agent, and the contract it was built against
web/        the serverless function and page behind the demo
evidence/   breaker verdicts and unedited transcripts

The Python harness and the serverless agent use only the Python standard library. The Studio uses
Sanity's normal React and TypeScript dependencies.

How I Used Sanity

The schema makes the distinctions representable; the endpoint instructions require the agent to
keep them separate; the harness independently enforces retrieval, required output fields, citation
syntax and named-object guards around the answer. It does not yet re-verify the returned values against the
document field by field — the instructions carry that, not the validator.

One honest wrinkle in my own design: the response VERDICT is a single enum that mixes claim state,
finding state, expiry metadata and retrieval outcome. That is a presentation-level classification.
The underlying status and expiryStatus stay separate in the dataset, and the graded claim lookup
reported both separately in its answer text — but the validator does not yet guarantee that
separation for every future answer, and a stricter contract would split the output the way the
schema does.

Three schema choices do the work:

1. status and expiryStatus are separate. A claim can be standing and no_expiry_set
at the same time. no_expiry_set means nobody dated it — not "permanently true," not "expired."
Collapsing those two facts into one is how a record quietly rots, so the schema represents them as
separate fields instead of forcing one value to stand for both.

2. A finding has commentOn and writtenUpIn. Where someone raised a defect and where I wrote
it up are different articles. pm25coder found B1 in the comments of one article; it was written up
in another. One field would have glued the wrong attribution to the wrong post.

3. patch.inMain is a boolean. Public is not merged. All three recorded patches are pushed and
none is in main. Asked about B1, the agent reports inMain: false rather than inferring merge
state from the branch being public.

Two instruments, because one couldn't do it

I built this on a Knowledge Base first. It handled the four prose-routed frozen questions in the
graded run. But I could not reliably recover a specific claim's status and expiry from the indexed
representation, and the measurement shows why:

all 24 Knowledge Base entries  = 190,503 characters
'claim-ledger-population'      = 1 occurrence — a label in a Sources list
its values                     = present, but not bound to that identifier

The values existed in the indexed prose but were not bound to claim-ledger-population. A prompt
cannot make that binding reliable when retrieval did not preserve it, so I routed that question
shape to GROQ — which is what Sanity recommends when the schema already says where to look.

I did not grade my own work here. Independent review means separate, owner-assigned AI reviewer
sessions that did not implement the candidate — not outside human testers. The first live graded
run they carried out exposed the wrong-object failure on exactly that question. Synthetic breaker
controls then found more: failed retrieval could reach the model, unsupported answers could exit
zero, parts of the output contract were not enforced, and the contracted tool-discovery step was
missing. The VERDICT.md in evidence/ is a
BLOCK and it stays there.

The fix was a second Context endpoint serving GROQ over the same dataset:

*[_id=="claim-ledger-population"][0]
→ status "standing" · expiryStatus "no_expiry_set" · asOf "2026-09-11"
  sourceUrl "https://dev.to/kenielzep97/my-harness-used-one-label-for-three-different-failures-2gc3"

One document, its fields, and its stored sourceUrl.

So the agent routes by question shape. Four of the five frozen questions — B1 attribution, B1
merge state, B8 receipts and Kubernetes — stay on the Knowledge Base. Only the exact claim-field
lookup routes to GROQ.

The router itself is blunter than that description: any question containing a claim-* token goes
to the dataset endpoint, which is broader than the rule I specified. The breaker preserved that as an open
remainder rather than letting me describe the abstraction as if it were the code.

By default, a Sanity Context endpoint's mode is derived from its configured sources: an endpoint
with a dataset source serves GROQ mode, and Knowledge Base sources on that endpoint are ignored. So
two endpoints, exactly one instrument per question.

self-correcting-systems       initial_context, knowledge_base_read
self-correcting-systems-data  initial_context, groq_query, schema_explorer, array_field_reader

Custom instructions on each endpoint carry the rules — report receipts not claims, keep status and
expiry apart, bound anything about mutable state to the snapshot date. They arrive through
initial_context. The harness fetches that context before letting the model read a Knowledge Base
entry or query the dataset, so the endpoint's instructions are in context before any evidence is
retrieved.

What it does not establish

The current corrected CLI harness held all five frozen questions on one independent v8 graded run
with gemini-3.6-flash. That is not general reliability. The earlier BLOCK used gemini-2.5-flash;
I am not claiming the same model failed and then passed.

One more limitation is preserved in the breaker verdict rather than smoothed over. The graded v6
candidate enforced the Knowledge Base citation rule incorrectly — it accepted an entry path or
the Knowledge Base id where the contract required both. The five live answers happened to supply
both, so their content held, but the guard was weaker than specified. Version 7 corrected that to
an AND and received its own independent run. A later review found that an unparseable VERDICT
could bypass the verdict-dependent checks. Version 8 rejects those malformed values and received
a new independent run: five of five, exit 0. Its CLI hash matches harness/ask.py in the repository.
The v6 and v7 transcripts remain evidence for their respective versions.

All four graded runs — the original BLOCK, v6, v7 and v8 — are in evidence/, unedited.
No earlier transcript was replaced to conceal its result.

Agent Session / Evidence Trail

I kept the failed run instead of replacing it:

• the original BLOCK and its unedited transcript
• v6 — five content-holding answers, weaker citation guard
• v7 — the AND fix, five of five, exit 0
• v8 — after a review seat found that an unparseable VERDICT skipped every check gated on it; five of five on the repaired file
• the latency receipt — five anonymous live requests and their measured timings

The historical verdicts remain unedited. GET /api/ask reports the sha256 of the deployed function,
so you can check which source is actually serving rather than trusting that a push landed.

Sanity Project Details

Project ID       u58x3mt0
Dataset          production (public)
Knowledge Base   kbjnxAgyAimV
Studio           https://self-correcting-systems.sanity.studio/

Query the record yourself, no credentials:

https://u58x3mt0.api.sanity.io/v2025-08-15/data/query/production?query=count(*)

*[_type=="finding" && status=="unbuilt"]{code, title, "who": foundBy[]->handle}

No login is required for the demo or the dataset.

I'd rather ship an agent that says two of fourteen are implemented, one of three B8 receipts is
locatable, and this record cannot see past its snapshot than one that turns every stored assertion
into certainty.