My Japanese Tutor Agent Works — Except Its Brain Calls Everything a Particle

I built a voice-first Japanese tutor that listens, corrects, scores pitch accent, and remembers your mistakes. The pipeline works end to end. The evaluation says the correcting brain collapses to one error type out of four. Publishing the failure is the point.

Scope note, first paragraph as promised: 300 sentences from Tatoeba (CC-BY) with script-injected errors — no fluent reader checked any label, disclosed throughout; one local Swallow-8B model, greedy decoding; simulated mistake histories, never real learners; no LLM judge anywhere.

The thing I built

Kotoba-Tutor is a LangGraph agent: push-to-talk CLI → faster-whisper transcription → Swallow-8B correction analysis (structured JSON) → pitch-accent scoring with my earlier Akusento model → short explanation → a cross-session store counting your mistake types → a scheduler drilling your weakest one. The explanation is spoken back with a local TTS voice. Everything runs on one RTX 3060; a VRAM spike up front (M0) decided the layout — LLM plus ASR on GPU, TTS on CPU — before anything else was built.

The loop genuinely works. I spoke an error sentence through TTS, fed the audio in: whisper transcribed it, Swallow flagged が→は with the exact gold restoration, accent scored 0.5, and the explanation came back as audio. That part is real.

The dataset: exact labels by construction

Without a fluent checker I couldn't trust human labels, so I inverted the problem: take 150 correct Tatoeba sentences as gold, inject one error each into 150 more (37 particles, 37 conjugations, 37 politenesses, 39 word choices) with a morphological script. Every gold tag is exact because the script put it there. The limitation is stated in the repo: some injections are defensible variants, which biases against the tutor. All 300 are on LangSmith and Hugging Face.

The result: type collapse

Pre-registered eval, memory vs memoryless, same backend: false-correction 0.92–0.94, recall 0.92+ on particles and 0.0 on the other three types. The confusion matrix is the whole story — the model predicts "particle" on 285 of 300 rows and never emits the other types. The memory reminder (your history: 12 particle mistakes…) changed nothing: a valid null. And the false-correction rate is an upper bound, because the "correct" gold rows are unverified crowdsource — on 4 trivially correct sentences the model passes 4/4, so it's harshness on real text plus unverified gold, not total inability.

This is the honest outcome and the reason for the pre-registration: the loop works, the brain needs a v1.1 (constrained type output or a classifier head, not free-form JSON). I'd rather ship a measured failure than an unmeasured demo.

Update: human gold confirms it (v1.1)

After this draft was queued, the corpus access came through (same-day replies from both labs — thank you Komachi lab and NAIST). I rebuilt the eval on 1,288 TEC-JL sentences with dual human references: 812 verified-correct plus single-type errors, same frozen prompt, same model. False-correction went up to 0.95 [0.93, 0.97] — the "unverified gold" excuse is dead; the model flags correct Japanese ~95% of the time. The collapse replicates exactly (particle ~1,240/1,288, other recalls zero, stray types like "structure" leaking out). Row texts are research-only and stay off Hugging Face; only the rates are published. The v1.1 brain fix is still the next step.

Update 2: the supervised head copies everything (v2)

I trained the obvious fix — LoRA on 50k real learner corrections — and it failed in the opposite direction: false-correction 0.95 → 0.09, but exact-match 0.14 → 0.01. The adapter copies the input nearly every time. Of course it does: every training example was has_error:true with tiny edits, so copying minimizes loss, and it never saw a correct sentence. Two measured failure modes now bracket the problem (trigger-happy base, copy-everything adapter); the fix needs correct examples plus loss that weights changed tokens. That experiment is queued, not claimed.

Update 3: restraint without repair (v3)

Ran it: 10k correct examples plus 4x loss on changed tokens. False-correction 0.95 → 0.17, exact-match 0.01 → 0.03 — passes my pre-registered bar on paper, but the model mostly learned to flag nothing at all. Three adapters, three policies (flag everything, copy everything, flag nothing), none of which corrects. The correction problem is harder than the plumbing around it.

What's next

Human-corrected gold (TEC-JL / NAIST Lang-8, both request-gated) is the v1.1 eval path; the v1.1 brain fix; the owner's recorded demo session. Code, data, and the pre-registration are linked below.