This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

I run RimWorld 1.6 with 232 mods. Getting that list to load cleanly took most of June, and the hard part wasn't finding answers. It was that the answers disagree. A mod's author declares its dependencies and load order in About.xml, except when they don't, or when the declaration is three versions stale. The RimSort community keeps a big set of load-order rules that fill those gaps. And then there's what I found out myself on this exact list: which log lines are noise, which mods I had to switch off, and what fixed what.

So I built an agent that reads your ModsConfig.xml and Player.log, checks the list against all three sources, and tells you what's wrong, who says so, and the smallest change that fixes it. When the sources disagree it's supposed to say who claims what instead of quietly picking one.

The test I cared about: in June I disabled nine mods, each for a reason I wrote down. I switched all nine back on, launched the game, took the real Player.log it wrote, and asked the agent "why does my game throw errors?"

Demo

npm run agent -- --list ../data/demo/ModsConfig.xml --log ../data/demo/Player.log "why does my game throw errors?"

The log is 300 KB. The log tool strips stack frames and collapses repeats, which leaves 49 distinct problem lines out of 363. The agent made 10 tool calls over 11 turns, and the run cost $0.36.

It found all nine mods and tied each one to the log line that gives it away. A few rows from its answer (the whole thing):

Mod (packageId)	Owner's reason (disabled[])	Log evidence
While You're Up (zsbk.patch16.whileyoureup)	Harmony patches fail on 1.6; PUAH covers it	Error while instantiating a mod of type WhileYoureUp.Mod: ... HarmonyException ... WorkGiver_ConstructDeliverResources
Smart Speed (sarg.smartspeed)	Harmony patches fail on 1.6; built-in speed controls cover it	Error in static constructor of SmartSpeed.Main ... Invalid IL code ... TimeControls::DoTimeControlsGUI
SimpleCameraSetting (ray1203.simplecamerasetting)	Conflicts with Camera+	check_load_order flags it incompatible (author) with active brrainz.cameraplus; matches incident incident-camera-conflict

It also found nine load-order violations and wrote a reordered ModsConfig.proposed.xml next to the original that fixes them by moving nine mods. It never touches your real file. There are 299 "Fallback handler could not load library" lines in that log, and it gave them one line:

Harmless, no action needed: Fallback handler could not load library ...dll (299×, Unity/Mono noise under heavy Harmony patching, confirmed harmless by the owner's incident log).

And it listed the handful of lines it couldn't pin to a mod, and said so, instead of guessing.

It got one thing wrong. It called the XML Extensions / Winston Waves order rule an "owner incident". That one comes from the RimSort community, not from me. The full answer and every tool call are in runs/.

Code

TypeScript on the Claude Agent SDK, Sanity Studio for the schema, and a Python script that builds the dataset from the mods installed on my machine. 17 tests, and npm run check works without any token because the dataset is public.

How I Used Sanity

The content

The dataset has 462 documents: 300 mods, 150 community rules, 10 incidents and 2 mod lists. There are four document types.

A mod is what its author declared in About.xml: supported versions, dependencies, loadAfter, loadBefore, incompatibleWith, plus a supersededBy reference when an old mod has a 1.6 fork. A rule is one RimSort community rule, pointing at the mods it's about. An incident is something I observed: symptom, the exact log signature, cause, fix, and a verdict. A modList is my list in load order, plus the mods I disabled and why.

Keeping those as separate types instead of flattening everything into mod descriptions is what lets the agent say where a claim came from. That's the whole point of the thing. If the author says A before B and the community says B before A, that's a contradiction finding with both sources attached, not an order error.

Two endpoints

The agent gets two Context MCP endpoints. rimworld-kb is the knowledge base, and rimworld-data does GROQ over the whole dataset. They're separate because an endpoint with a dataset source only serves GROQ mode.

The knowledge base is built from one GROQ query, since a KB takes one query per dataset. It's a union of all 10 incidents, the 51 community rules that have a note and touch a mod in my list, and 36 mod descriptions that actually explain something. That comes to 97 documents, and the reason it isn't all 462 is the 150 indexed document cap during the beta. Choosing what goes in was more work than the query.

The KB earned its keep before the agent ever ran. While building entries, Context flagged two conflicts on its own. One entry claimed every mod in the UI section requires Harmony, and it pointed out that Holsters' About.xml declares no dependencies at all. The other was about Pocket Sand and Combat Extended, where the author declares Pocket Sand loads first. I resolved both, and both became eval questions.

A local tool for the part a model shouldn't eyeball

Checking load order is a graph problem, and I didn't want a model working out 241 positions by reading a list. check_load_order runs locally as a third MCP server. It pulls the relevant documents from the public dataset, checks every claim, and returns findings tagged with their source, plus the smallest reorder that satisfies all of them. The agent reads the KB to explain why a finding matters and what fixed it last time, and uses GROQ for exact facts.

This is the checker on its own against the demo list, which is also what you get from npm run check without any token:

Does the knowledge base actually help?

I wrote 15 questions with reference answers from the dataset, including one it has no data for (Rimatomics vs Save Our Ship 2, where the right answer is "not covered"). Same model (Claude Sonnet 5), same prompt, three conditions: no tools, the local tools plus GROQ, and all of that plus the KB. Claude Opus 5.5 graded every answer against the reference in shuffled order without knowing the condition.

The agent read the KB in 12 of the 15 answers where it had one. The questions it flipped were the ones where the reason matters more than the fact. With only GROQ, asked whether SimpleCameraSetting can run with Camera+, it suggested disabling Camera+, which is the one I kept. Asked about stack traces under every line on 1.6, it blamed the Fallback noise. The KB has the incidents for both.

Some honest caveats. It's one run per question. I wrote the questions from the same data the agent reads, so the no-tools row mostly shows what a model can't know about my list, and I wouldn't read it as a benchmark of the model. The judge can sometimes tell a tool-using answer from its wording. Two of my references (VFE research, Filth Creates Trash) assumed the mod was missing, but the demo list already has it, so correct answers saying "it's already in your list" got marked down. I fixed those two references and regraded. Before the fix the three rows were 2, 9 and 12 correct, and both versions of the grades are in the repo along with every answer.

The one miss in the last row is real. Asked about the old Fluffy Breakdowns, it mixed it up with the 1.6 fork that's already in the list and never followed supersededBy to the replacement. Every condition missed that one.

What didn't work at first

The first demo run said VPE when it meant VEE and got one load direction backwards. Findings used to be bare packageIds like vanillaexpanded.vee and it was guessing at names. Now each finding carries the mod's title and states the requirement before the problem ("X should load after Y, but loads before it"), and both mistakes went away.

The checker flagged Prepatcher for sitting above a load-top mod. Prepatcher declares it loads before the game itself, so no load-top rule can outrank it. That's fixed and there's a test for it.

Asked to write the proposed order to a relative path, the tool wrote it into the agent's working directory while the agent told me it was next to my mod list. Relative paths now resolve against the list.

And the 299 Fallback lines didn't collapse at first, because the addresses in them are hex without a 0x prefix, so every line looked unique.

Sanity Project Details

• Project ID: 49jweiga
• Dataset: production (public)
• Try it: https://49jweiga.api.sanity.io/v2025-02-19/data/query/production?query=*[_type=="incident"]{title,verdict,fix}
• Studio: https://rimworld-loadorder.sanity.studio/ (the schema is in the repo under studio/)