Five bugs in my LLM app that never threw an error

I spent two and a half weeks building an agent that reads a project's code review history and remembers what it decided. 86,321 comments from pandas, distilled into 298 conventions, served out of CockroachDB behind a Lambda.

Every serious bug I hit had the same shape. Nothing raised. Tests passed. The demo worked. And somewhere in the middle, a component was doing absolutely nothing.

Here are five, with the code.

1. bool("false") is True

The whole point of the system is that it refuses when it does not know. The answering prompt returns JSON with an answered field.

result = await chat.complete_json(messages)
if result.get("answered"):
    return result["answer"]

A model that writes "answered": "false" instead of "answered": false turns a refusal into an answer, because a non-empty string is truthy. The one behaviour the project exists to guarantee, defeated by a quotation mark.

Now every model response goes through a Pydantic model that fails towards doing nothing:

class AnswerOutput(ModelOutput):
    answered: bool = False   # coerces "false" -> False
    answer: str = ""

An answer becomes a refusal. A drafted rule becomes unusable. A contradiction verdict becomes "compatible", which retires nothing. A garbled response is not evidence for writing to memory.

2. A null discarded whole responses, six times out of six

The fix for bug 1 caused bug 2, which is funnier than it was at the time.

The prompt that turns a maintainer's comment into a rule says:

"rationale": "why, one sentence, only if they gave a reason"

A maintainer who states a convention flatly gets null back, which is correct JSON for "no reason given". And:

class DraftedRule(BaseModel):
    statement: str = ""
    rationale: str = ""   # null is not a str

Pydantic rejects null for a plain str. Validation is all or nothing. So the entire response was thrown away and replaced with the inert default, which downstream reads as the model having declined.

A real maintainer correction became {"status":"ignored","reason":"no convention stated"}. Six reproductions, six failures, on a payload that was otherwise exactly what I asked for.

class ModelOutput(BaseModel):
    @field_validator("*", mode="before")
    @classmethod
    def _null_text_is_empty(cls, value, info):
        if value is not None:
            return value
        field = cls.model_fields.get(info.field_name)
        return "" if field is not None and field.annotation is str else value

A null where a string was expected means empty, not malformed. Note the field.annotation is str check: scope_pattern: str | None genuinely means None, and coercing that one would turn "applies everywhere" into a pattern matching nothing.

3. 68 of my 79 path rules matched zero files

Each rule stores a pattern for which files it applies to. I matched with fnmatch.

fnmatch("pandas/core/frame.py", "pandas/core/%")   # False

The patterns were SQL LIKE, not globs. pandas/tests/%, pandas/core/%, pandas/tests/%/conftest.py. Nothing asked the extraction prompt for that and nothing documented it. The model just wrote it, presumably because the rules were destined for a database.

% is not an fnmatch wildcard. So 68 of 79 anchored rules silently matched nothing, and the agent ran entirely on the two rules that happened to be written pandas/**/*.py, which cover 63% of the repository.

It looked like it was working the entire time. The fix is one line, and the bug was invisible for a week:

cleaned = pattern.strip().strip("`").lstrip("./").replace("%", "*")

4. The similarity score I was tuning had no signal in it

The agent comments on pull requests, choosing which conventions apply by embedding a description of the change and taking the nearest rules.

It commented on 38 of 40. I assumed a bad threshold and started tuning. At 1.05 it spoke on 95% of pull requests; at 0.95, on 18%.

Then I measured the distribution instead. For 25 real pull requests, distance to the nearest rule versus the tenth nearest:

The nearest rule is not meaningfully nearer than the tenth, under any phrasing. Every rule is a general statement about the same codebase, so every rule is similar to any description of a change to it.

There was nothing to threshold. Retrieval now runs on path matching, which is a fact, and similarity only orders what the paths already admitted.

5. GitHub's author_association describes the present

I filtered to maintainer comments using GitHub's author_association field.

It reports whether someone has write access now, not when they wrote the comment. One reviewer wrote 74,077 comments on pandas between 2012 and 2025, more than twice anyone else. He has since left the org, so all 74,077 come back as CONTRIBUTOR.

Filtering on that field discarded about a third of the corpus, including the most experienced reviewers the project ever had. No error, no warning, just a smaller number that looked plausible.

The pattern

None of these raised. Four of the five I found by counting something rather than by reading code: how often does it speak, how many rules match anything, how far apart are the distances really.

A library used wrong fails loudly. A model used wrong returns confident, well-formatted output that is quietly detached from what you meant.

So pick a number that should hold if the system works, and go and look at it. Not "does it return results" but "how often does it speak, and is that the rate I intended". Not "did it cite something" but "do the citations resolve".

Code is at github.com/pyarchana/precedent, MIT.