Sparse autoencoders find human-readable features inside LLMs. I trained one per layer on a Japanese-English model and measured which features fire in both languages — then tested whether they actually matter, with controls. The correlation is beautiful. The causation is borderline. Both get reported.
Scope note, first paragraph as promised: one model (llm-jp-3-1.8B), hand-rolled TopK SAEs (the standard library doesn't support this architecture), 2M training tokens per layer, Tatoeba pairs for analysis, greedy decoding, pre-registered causal tests with random controls. No human labelled any feature — only the intervention numbers carry weight.
The question
Every Japanese AI company building bilingual systems needs to know: does the model represent 契約 and "contract" with one shared feature or two separate ones, and at which depth do languages merge? Most interpretability work studies English-heavy models. I picked llm-jp-3 — genuinely bilingual pretraining — and measured.
The microscope (and its calibration)
One TopK SAE per layer (4/12/20; 2048→32,768 features, 64 active), trained streaming on Wikipedia Japanese+English. Quoted numbers are held-out only: early-layer death 0.27 with resampling (without it, 0.59 — in-train dead counts of 0 are meaningless, a lesson burned into the decisions log), reconstruction costs +0.28 to +0.31 nats of LLM loss. Late layer is hardest (+0.74, undertrained — its results below carry that caveat explicitly).
The ladder: names, then concepts, then script
On 20,000 parallel sentence pairs, per-feature paired correlation: the share of cross-lingual features rises 0.25 → 0.29 from early to middle, then falls to 0.13 late. And the top features tell a story words alone can't: early layers detect NAMES (Mary/Maria, r=0.99), middle layers detect CONCEPTS (burning — the house burned down and 家は燃え尽きた fire the same feature), late layers mix concepts with script features (the kanji 英 prefix fires on Shakespeare, 英作文, 英知 — English-related, but by spelling, not meaning). A static browser with all top-100 features per layer ships with the repo — go argue with my labels, that's what it's for.
The causation test (pre-registered, mostly null)
Correlation isn't causation, so: ablate 30 shared concept features and measure next-token probability shifts in both languages, steer them and check cross-lingual movement — each against 30 random features on identical prompts, 95% CIs. Result: ablation moves Japanese slightly beyond random (−0.046, CI excludes zero) on top of massive generic damage (−0.48 vs −0.44 — breaking SAE reconstructions hurts everything), English is null (−0.010, CI includes zero), steering overshot off-manifold and is uninformative by prior admission. Verdict: weak causal evidence in Japanese only. The ladder stands; the lever bends.
What I'd do with 10x the compute
Bigger SAEs (the 0.59→0.27 death gap says capacity and resampling matter), a retrained late layer, and steering with calibrated magnitudes. All code, weights, and numbers are linked below; all three SAEs are on Hugging Face, Apache-2.0, reproducible by anyone.
