The rename compiled clean, and the error still lied
We asked the agent to rename an internal service class after a domain-term change. It swept the symbol graph: class definition, every import, type annotations, the constructor, the call sites, even the renamed test. The module compiled, the suite passed, CI went green, and we merged it.
Three days later a user hit the failure the class guards against. The stack trace and the unit test looked coherent, but the thrown message still carried the old name nobody used anymore. On the support thread, a support engineer grepped the current term, found nothing, and spent an hour hunting a component that had not existed since the rename.
What actually happened
The agent modeled the rename as a graph walk over identifiers. Every edge that referenced the symbol got updated, because that is what a rename operation actually is: follow references in the compiler's symbol table. It stopped there.
The old name also lived in places the symbol table does not own: the free-text message inside the thrown error, a log tag attached to the same code path, a localized UI label, a metric name, and a route description shown in the internal admin panel. None of those are identifiers, so a rename pass never walks them, and the compiler has no reason to flag a string that still spells the old word.
The tests did not catch it because they assert on the error code, the thrown type, or the branch taken, not on the free-text copy. Green tests proved the guard fired. They said nothing about what the message said.
Why the agent stops at the symbol table
Renames feel complete once the build passes. A compiler checks that identifiers resolve; it cannot check that a human-facing sentence still matches the current vocabulary. The agent treats a clean compile as the finish line, because that is the strongest signal the toolchain offers and it is readily available.
Strings look like data, not code. To the agent, quoted text in an error template or a locale file sits outside the code being renamed. Editing it feels optional, decorative, and easy to skip under a tight task frame.
The task said "rename the class," not "rename the term." Handed a symbol name, the agent renames that symbol. Nobody expanded the scope to the vocabulary the symbol represents, so display copy was never in the frame.
The drift that survives a green build
This pattern reaches well beyond one error message. A renamed feature keeps its old internal metric name, so the dashboard keeps plotting under the legacy label and nobody sees the new traffic. A renamed resource leaves an old route description in a generated admin panel. A localized label file keeps the previous term while the code around it uses the new one. In each case the code is correct, the tests pass, and the words the human reads are stale.
Free-text messages are also the worst place to assert against. They drift freely, get reworded during a polish pass, and break a test the moment someone edits a sentence. That is exactly why tests target a stable error code rather than the whole sentence — but the stable code still needs its documented human-facing copy checked somewhere.
The checks that catch what the compiler compiles past
Grep the old token across every file type, not just code. After a rename, search the legacy term in JSON, YAML, locale files, error templates, seed data, route metadata, and docs, not only .ts/.py sources. The symbol walk never enters those files, so a plain whole-repo grep for the old word is the safety net.
Audit user-facing strings as a separate pass. Treat display copy, log tags, metric names, and thrown messages as data that belongs to the renamed concept, not a footnote to the identifier change. List them deliberately rather than hoping the rename reaches them.
Assert on a stable code, not the sentence. Key errors by a stable machine-readable code and assert the code in tests; then review the accompanying human copy against a spec or copy sheet separately, so rewording the message does not break the suite and drift still gets caught.
Snapshot labels and dashboard names when a rename touches them. UI labels and metric names are user-visible state; a renamed entity whose old label still appears in a snapshot fails review instead of shipping unnoticed.
The contract version of the same drift
Public APIs carry this risk most sharply. A renamed field or operation leaves its old example message, error string, or description in the public surface, while the code and internal docs have moved on. Prose in a README cannot enumerate every live message, so the copy drifts unless it is checked against the contract itself. We keep that loop local in Powerduck: the OpenAPI spec holds the documented error codes, example messages, and operation descriptions, and a live diff against the endpoint flags a renamed operation whose public copy was left behind.
What I check before I call a rename done
- Did the grep for the old token reach strings, locale files, templates, and metadata — not only identifiers?
- Is there an audit list of thrown messages, log tags, UI labels, and metric names tied to the renamed concept?
- Do tests assert a stable error code instead of a free-text sentence that drifts?
- Was the public copy reviewed against the contract rather than trusted to a symbol rename?
A rename that reaches every identifier can still leave the old word in everything a human reads. The build proves the symbols resolved; it never proved the words kept up.
