Dropping Candidates Due to Fixable Defects — A Story of Measuring Factory Flaws as Product Personality

📝 Originally published (in Japanese) at forge.workstyle.tech.

I Built a System to Mechanically Select Voice Model Candidates

I created a system to mechanically select from voice model candidates. It reads probe sentences with 24 candidate voices, automatically measures and scores them based on Whisper match rate, vowel elongation, speech rate, intonation, and jitter, then selects the top candidates for manual review.

The system allows weighting to be adjusted based on role. For narrators, slower speech rates are given positive weight, while for MCs, wider intonation ranges are favored. As a common penalty, Whisper match rate was weighted between 1.0 and 1.2. Candidates that couldn't read the script accurately were eliminated. This seemed reasonable at the time.

It turns out this was fundamentally wrong in terms of metric design.

There Was a Role with Abnormally Low Match Rates

When generating candidates for presenters, the match rate was significantly different. While other roles had rates of 0.93–1.00, presenters only achieved 0.70–0.93, with greater variability in speech rate.

Upon inspection, hallucinations appeared at the end of generated speech:

Original: なぜだと思いますか。
Transcription: なぜだと思いますか?そうだ!これに似てる円形と思いました

Original: ご清聴、ありがとうございました。
Transcription: ご清聴ありがとうございました。元気をございました。

When short sentences (9–14 mora) were combined with emphasis captions ("with good pacing, emphasizing important points"), extraneous speech was added. When the probe sentences were rewritten to 18 mora or longer, the match rate returned to 0.93–0.96, and the top ranking changed.

Initially, I thought "the measurement was broken and needed fixing." But the fact that rankings changed was concerning. If a measurement error could change the top ranking, that meant the metric was effectively determining candidate superiority. So what exactly was this metric measuring?

Revisiting the Pipeline Structure

This system uses two different engines:

[Design Phase] Diffusion TTS → Generate voice with caption+seed → Training corpus (~200 samples)
                                                      ↓
                                                    Training
                                                      ↓
[Runtime] Trained model ──────────────── Actual voice heard by users

Diffusion TTS is specialized for voice design. The voice is determined by the caption and random seed, producing identical voices from the same combination. However, generation is slow (RTF 0.25–0.54) and unsuitable for conversation. Therefore, the generated audio is used as training material to bake a lightweight model, which handles runtime playback.

This means hallucinations occur in the diffusion TTS stage—the corpus creation process—while users actually hear the trained model—the product. These two are different things.

Do Trained Models Hallucinate?

I prepared six short sentences that always produced hallucinations in diffusion TTS and trained three models from them.

Diffusion TTS:       "なぜだと思いますか?そうだ!これに似てる円形と思いました" (transcription length 2.9x)
Trained model:       "なぜだと思いますか" (ratio 1.00)

With the same 9-mora sentence, one version fabricates content while the other is clean. Hallucinations are a defect of the manufacturing process and don't appear in the product.

We were rejecting candidates for a fixable flaw. Worse, the flaw doesn't even exist in the product.

What the Metrics Were Actually Measuring

The scoring formula looked like this:

Score = 1.0 × match_rate − 0.25 × vowel_elongation − 1.0 × speech_rate_cv − 2.0 × jitter ...

I thought match rate measured "whether the voice reads the script accurately." In reality, it measured "whether this caption+seed combination is less likely to hallucinate with short sentences." This was a measure of process state, not voice quality.

Worse, hallucinations contaminated speech rate measurements. Added speech extended the duration, making mora/second appear slower than actual. After fixing the probe, speech rate changed from 4.0–6.3 → 5.2–7.2 mora/second, converging to the target range. A single defect was distorting multiple metrics.

What was the result? The top female presenter candidate was seed 864200, but after fixing the probe, seed 123456 rose to first place. Seed 123456 had an intonation range of 26.0st, significantly larger than other candidates (11.7–18.3st). The candidate with the strongest "emphasis variation" needed for presentations was ranked third due to hallucinations.

Searching for Reasons to Keep the Metric

At this point, I considered:

In presenter roles, short emphasis phrases are spoken during actual operation. So isn't it valuable to select voices that are less likely to hallucinate with short phrases?

This seemed plausible but didn't hold up. In actual operation, the trained model is used, where hallucinations don't occur. The property "weak with short phrases" doesn't exist in the product.

Realizing this made me understand that I didn't want to abandon the metric. I was trying to justify the system I'd built.

However, "Not Inherited" Was an Overstatement

The story continues. If trimming is used to remove hallucinations before training, hallucinations shouldn't appear in the corpus at all. So were we training with hallucination-containing samples?

I re-examined the script matching code:

def judge_transcript(script_text, transcript,
                     min_ratio=0.40, max_inserted=3):
    """Match script with transcription. Reject if insertions (text not in script) > max_inserted."""

Insertions of up to 3 characters are accepted. Insertions of 4+ characters (e.g., "たふにボンジュそのごう。" = 11 insertions) are rejected, but insertions of 3 or fewer characters are included in the training material.

When these were read by the trained model, the following occurred:

つまり、こういうことです。       → "つまり、こういうことです。ヴェー"   (2 insertions)
大丈夫ですよ、気にしないでください → "大丈夫ですよ。気にしないでください。いや。" (1 insertion)

The size of what passed the gate matched the size of added speech reproduced by the model.

Further inspection using waveform envelopes revealed defects not visible in STT-dependent inspection (there are defects transcription doesn't catch):

ご覧ください。   Original 0.88s → Silence 0.48s → [0.16s speech]   STT: "ご覧くださいああ"
こちらです。     Original 0.72s → Silence 0.56s → [0.28s speech]   STT: Not captured

Speech separated by 0.5 seconds of complete silence cannot be a trailing reverberation. "0/6 didn't carry over" only meant it didn't happen in those six sentences—we hadn't confirmed it was sentence-dependent. Even though I'd observed "ヴェー" myself, I'd written it off as "minor noise" and generalized from there.

The correct understanding is:

• Large fabrications aren't inherited (the gate rejects them)
• Small insertions that pass the gate are reproduced by the model (quality gate's allowed "3 characters" became the model's verbal tic)

Therefore, removing match rate from selection metrics is correct, and simultaneously "tightening manufacturing gate tolerance" is also correct. Two separate layer issues were overlapping.

Fix 1: Trim Before Measurement

On the selection side, hallucinations are removed before metrics are measured. Just changing the order:

# Before: Generate → Measure
wav = gen(text, caption, seed)
ratio = judge_transcript(text, whisper(wav)).ratio   # Value including hallucinations

# After: Generate → Trim hallucinations → Measure
wav  = gen(text, caption, seed)
segs = transcribe(wav)["segments"]
end  = script_end_sec(text, segs)          # Position where script ends
wav  = trim_to_segment_end(wav, end) if end else wav
ratio = judge_transcript(text, whisper(wav)).ratio   # Only voice properties remain
speech_sec, level, f0, range_st = acoustics(wav)     # Speech rate & F0 not contaminated

Now match rate reflects only "sentences that truly can't be read, even after trimming." As a side effect, speech rate contamination also disappears because added speech no longer inflates duration.

At the same time, we no longer need to worry about probe sentence length. Before the fix, we had a warning in the screener: "Don't use probes shorter than 15 mora as they invite hallucinations," but if trimming works, that constraint isn't needed. Getting rid of a workaround was surprisingly significant.

Fix 2: Tighten Manufacturing Gate Tolerance

Reduce insertion tolerance from 3 to 1. However, doing this directly would reject normal clips, so we first need to absorb notation variations.

Long sounds are just notation differences

This insight was key:

Script: ふふっ、今日は最高に楽しい一日でした！
Transcription: ふっふっ、今日は最高に楽しい一日でした!     → 2 insertions

Script: ど、どうしよう。胸がざわついて…
Transcription: どっどうしよう 胸がざわついて…            → 2 insertions

Both are just differences in whether the same sound is written as っ or 、. Existing normalization dropped punctuation but kept long sounds, leaving characters on one side that were counted as insertions. Now we also drop long sounds during comparison.

_PUNCT_RE  = re.compile(r"[、。！？!?…・\s「」ー〜,\.]")
_REPEAT_RE = re.compile(r"(.)\1+")
_SOKUON_RE = re.compile(r"[っッ]")        # Added

def _collapse(s):
    return _REPEAT_RE.sub(r"\1", _SOKUON_RE.sub("", _PUNCT_RE.sub("", s or "")))

"ふふっ" → "ふふ" → (repetition compression) → "ふ", "ふっふっ" → "ふふ" → "ふ". They match.

We also verify that unwanted items remain. "〜です。ヴェー" stays at 2 insertions, "〜たふにボンジュそのごう。" stays at 11 insertions. We list both items we want to save and those we want to reject to ensure normalization doesn't let the latter through.

Kanji/number notation variations absorbed via reading

Whisper doesn't necessarily transcribe using the same notation as the script:

Script: ここが、今日いちばん大事なポイントです。
Transcription: ここが今日一番大事なポイントです

Script: 月々のコストを三割ほど抑えられます。
Transcription: 月月のコストを3割ほど抑えられます

Existing "kana normalization" only converted katakana to hiragana, leaving kanji and numbers unchanged. Now we convert both sides to readings using pyopenjtalk's g2p:

_G2P = None

def _reading(s: str) -> str:
    # Absorb kanji/number notation variations by converting to readings (kana).
    # Fall back to surface comparison if dictionary isn't available to avoid regression.
    global _G2P
    if _G2P is None:
        try:
            import pyopenjtalk
            pyopenjtalk.g2p("あ", kana=True)      # Check dictionary availability
            _G2P = pyopenjtalk
        except Exception:
            _G2P = False
            logger.warning("pyopenjtalk dictionary unavailable, disabling reading normalization (surface comparison)")
    if _G2P:
        try:
            return _G2P.g2p(s, kana=True)
        except Exception:
            return s
    return s

def judge_transcript(script_text, transcript, min_ratio=0.40, max_inserted=1):
    a = _kana(_collapse(_reading(script_text)))
    b = _kana(_collapse(_reading(transcript)))
    ...

g2p('ここが今日一番大事なポイントです')      → ココガキョーイチバンダイジナポイントデス
g2p('ここが、今日いちばん大事なポイントです。') → ココガ、キョーイチバンダイジナポイントデス。

Even with different notation, readings match.

The trap of working without a dictionary

This is where I got stuck. pyopenjtalk may have the library installed but doesn't include the dictionary, attempting to download it on first use. In production containers, site-packages is read-only, causing this to fail:

PermissionError: [Errno 13] Permission denied:
  '/opt/venv/lib/python3.11/site-packages/pyopenjtalk/dic.tar.gz'

This silently falls back to the fallback path, leaving reading normalization ineffective while operating normally. In the image build, expand the dictionary while root has write permissions:

# Expand pyopenjtalk dictionary during build. At runtime, site-packages is
# read-only, causing initial download to fail with PermissionError, disabling reading normalization.
RUN python -c "import pyopenjtalk; print(pyopenjtalk.g2p('テスト', kana=True))"

USER fastapi

After deployment, verify in the Pod that the dictionary is working. If _reading() returns the input unchanged, the dictionary is missing:

>>> verify._reading('ここが今日一番大事なポイントです')
'ココガキョーイチバンダイジナポイントデス'      # If kana is returned, dictionary is working

Set thresholds using real data

Don't decide to "reduce from 3 to 1" based on intuition. Re-transcribe existing training material to count how many would be lost. Audio and scripts are preserved, so this can be measured without regeneration:

# Re-transcribe 75 existing clips to compare old vs new judgments
for clip in sample_clips(75):
    wav   = download(clip.r2_key)
    trans = transcribe(wav)["text"]
    ins_old, ratio_old = measure(clip.text, trans, normalize="old")
    ins_new, ratio_new = measure(clip.text, trans, normalize="no_sokuon+reading")

Average ratio improved from 0.963 → 0.995. With ≤1, no false rejects while catching insertions of 2+ characters.

Reducing to ≤0 would reject 3 clips, but all were Whisper-side issues:

Script: 本日の予定はすべて滞りなく進んでおります。
Transcription: 本日の予定はすべて[とどこおり/届こおり]なく進んでおります

The synthesized voice pronounces correctly, but transcription notation causes rejection. Decided ≤1 was appropriate since capturing ASR variations would be excessive.

⚠️ This measurement has bias. The sample only includes "clips that passed and were saved," so it answers "how many would be lost if tightened" but not "how many could be saved if loosened." Only useful for making thresholds stricter.

Still, this "re-judge existing assets to count losses" approach can be used every time thresholds are adjusted. Making arbitrary thresholds stricter causes yield to drop, and it can take weeks to notice. This method prevents that.

Questions to Ask When Creating Metrics

Is the metric measuring product properties or process state? In this case, match rate was entirely the latter. Fixing the process would eliminate this value, but it was being used to judge candidate superiority.

Only reject properties that can't be fixed. Process defects should be fixed in the process. Selection should only reject inherent undesirable properties of the candidate.

Metrics aren't independent. As hallucinations contaminated speech rate, a single defect can distort multiple metrics. When seeing correlations like "candidates with low match rates also have abnormal speech rates," we should suspect a common cause.

Are you writing conclusions that contradict your own data? When inconvenient observations are dismissed as "minor" and set aside, the conclusion is already broken. This is exactly what happened here.

Series: Mass-producing Practical Voices from Diffusion TTS

A record of designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 3: Quality Gates.

← Previous: Quality Gate's Allowed "3 Characters" Became the Model's Verbal Tic
→ Next: There Are Defects Transcription Can't Catch

Full series (18 parts)

1. TTS Chosen for Audio Quality Was Too Slow for Conversation
2. Rolling Voices Like a Gacha
3. Having the Machine Select "Narrator-like Voices" from 24 Candidates
4. The Harder the Quality Gate, the More Monotone Voices Survive
5. Speech Rate Can't Be Changed After Training
6. TTS That Changes "Recording Location" Every Time It's Generated
7. One Rough Clip Ruins the Entire Style
8. Where Did the AI's Habit of Elongating "konnichiwa~" Come From?
9. 「少々」Becomes「しょも」— The Allowed Character List Was Truncating Japanese
10. Hallucination Prevention Code Only Worked When Hallucinations Occurred
11. Quality Gate's Allowed "3 Characters" Became the Model's Verbal Tic 12. We Were Rejecting Candidates for Fixable Flaws ← Now reading
12. There Are Defects Transcription Can't Catch
13. 70 Minutes of Training Material Was Lost to a Network Blink
14. From "ja" to "JP": How a Babbling Model Was Born
15. Four Registration Paths, Zero Management Screens
16. Deploying Erased Each Other's Work Every Time
17. Pushing Unmeasurable Things with Thresholds Always Fails

The notebook containing the insights that led to this article is available in Diffusion TTS to Mass-produce Practical Voices Manufacturing Pipeline.