The '3-Character' Catchphrase That the Quality Gate Allowed Became the Model's Habit

📝 Originally published (in Japanese) at forge.workstyle.tech.

A trained speech model was asked to read a short sentence, and this is what it returned:

Input: つまり、こういうことです。
Transcribed output: 「つまり、こういうことです。ヴェー」

Input: 大丈夫ですよ、気にしないでください。
Transcribed output: 「大丈夫ですよ。気にしないでください。いや。」

There was an extra short sound at the end. It was a meaningless sound, 1-2 characters long.

The cause was the quality gate in the corpus. The size of the hallucination that passed and the size of the sound reproduced by the model matched.

Script Matching Gate Tolerance

In corpus generation, the audio read by TTS is transcribed by Whisper and matched against the script, and only the clips that pass are saved. The judgment is as follows:

def judge_transcript(script_text, transcript,
                     min_ratio=0.40, max_inserted=3):
    """Match the script and transcription. Clips with more than max_inserted (sounds not in the script) are disqualified."""
    a = _kana(_collapse(script_text))
    b = _kana(_collapse(transcript))
    sm = difflib.SequenceMatcher(None, a, b)
    inserted = sum(j2 - j1 for op, _i1, _i2, j1, j2 in sm.get_opcodes()
                   if op == "insert")
    if inserted > max_inserted:
        return VerifyResult(False, sm.ratio(), inserted, "Sounds not in the script inserted")
    ...

max_inserted=3. Clips with up to 3 characters not in the script are passed.

I remember the reasoning behind this value. Since Whisper's transcription is not perfect, some noise had to be allowed to avoid reducing yield. In fact, there was a problem where even normal clips were rejected if the criteria were too strict (because variations in notation were not accounted for, this was a correct observation).

However, I did not confirm that "3 characters would not cause any harm."

Size of Hallucinations That Passed

Diffusion TTS has a tendency to add extra speech at the end of short sentences. Here's a large example:

Script: 猫が窓辺で丸くなって眠っている。
Transcription: 猫が窓辺で丸くなって眠っている。たふにボンジュそのごう。   → Inserted 11

With 11 inserted characters, this is disqualified and excluded. This part was working (however, there was another bug in the trimming process for removal. The hallucination guard code only worked when there was no hallucination).

The problem is with smaller ones.

Script: 大丈夫ですよ、気にしないでください。
Transcription: 大丈夫ですよ。気にしないでください。いや。               → Inserted 1

"いや。" is 2 characters without punctuation, and 1 character after normalization. It passes and becomes part of the training material.

The corpus consists of about 200 clips per voice. A few of these had such short extra sounds mixed in, and the model learned the pattern of "adding a little sound at the end."

Reproducing in the Trained Model

After creating 12 voices, I had each read short sentences and checked them. Initially, I only looked at Whisper's transcription and judged as follows:

Female Presenter: 0 extra sounds in 6 sentences
Male Presenter: 0 extra sounds in 6 sentences
Female MC: 0 extra sounds in 6 sentences

I was about to conclude that it hadn't been passed on. But my measurement method was too lenient. Whisper drops short extra sounds. Speech lasting 0.1 seconds with a silent gap might not appear in the transcription.

When I remeasured using the waveform envelope, I got completely different results (the measurement method is detailed in "Defects Not Found in Transcription").

Please look here.   Main body 0.88s → Silence 0.48s → [Speech 0.16s]  STT: "ご覧くださいああ"
This way, please.     Main body 0.72s → Silence 0.56s → [Speech 0.28s]  STT: Not picked up

0.16 seconds of speech after a 0.5-second complete silence. This cannot be the tail of the main speech. After finishing the script, the model is saying something.

Mechanically Scanning 12 Voices

I measured how often this occurred across all voices. The judgment was "whether there is speech of 0.06 seconds or more after a silence of 0.25 seconds or more following the end of the main speech."

def segments(wav_bytes, thr_ratio=0.06):
    """Returns [(start sec, end sec), ...] of voiced blocks"""
    w = wave.open(io.BytesIO(wav_bytes)); sr = w.getframerate()
    x = np.frombuffer(w.readframes(w.getnframes()), dtype=np.int16) / 32768
    W, H = int(sr * 0.020), int(sr * 0.010)          # 20ms window / 10ms hop
    rms = np.array([np.sqrt(np.mean(x[i*H:i*H+W]**2))
                    for i in range(max(0, (len(x)-W)//H))])
    act = rms > max(rms.max() * thr_ratio, 0.004)     # Relative + absolute double threshold
    segs, s = [], None
    for i, a in enumerate(act):
        if a and s is None: s = i
        elif not a and s is not None:
            if (i - s) * 0.010 >= 0.03: segs.append((s*0.010, i*0.010))
            s = None
    if s is not None: segs.append((s*0.010, len(act)*0.010))
    return segs

def has_trailing_artifact(wav, gap_min=0.25, tail_min=0.06):
    segs = segments(wav)
    if len(segs) < 2:
        return None
    gap  = segs[-1][0] - segs[-2][1]     # Silence before the last block
    tail = segs[-1][1] - segs[-1][0]     # Length of the last block
    return (gap, tail) if gap >= gap_min and tail >= tail_min else None

The double threshold (rms.max() * 0.06 and absolute 0.004) is to prevent silent judgment errors in voices with low sound pressure. With only relative, everything becomes voiced in quiet voices, and with only absolute, even breaths become voiced in loud voices.

Results of scanning 12 models × 6 sentences:

8 out of 12 were clean, 4 had extra sounds. Contamination was not widespread, with significant differences depending on the voice.

When compared with the number of disqualifications during corpus creation, there was a correlation. Voices with more disqualifications (conditions where hallucinations frequently occurred) tended to have more small hallucinations that passed the gate.

Tightened the Threshold. It Didn't Work

On the creation side, the insertion allowance was tightened from 3 to 1. Simultaneously, normalization was introduced to absorb variations in notation (geminate consonants, kanji, numbers). Without this, normal clips would be rejected when tightened. After re-judging 75 saved clips and confirming no omissions with an allowance of 1, it was applied.

"ヴェー" is inserted 2 times, so it is rejected. The chance of mixing is calculated to be less than one-third.

After retraining and measuring, it was 2/6. Only one less than 3/6.

Thinking that tightening further would eliminate it, I set it to 0. This was a straightforward decision, and I had already measured that 3 basic and 3 conversational clips would be disqualified out of 203 actual clips. Since the basic disqualifications were exactly at the limit, I raised the limit from 3 to 6 and combined it.

After retraining and measuring, it was 3/6. It increased.

The difference between 2/6 and 3/6 might be due to measurement variation. But at least, "setting it to 0 did not eliminate it." Moving the threshold three times, the effect was within the margin of error.

Scanning the Material in Waveform Revealed Everything

At this point, I finally looked at the corpus material itself. I applied the detector used for the trained model to 198 clips.

Scanned 198 clips
  Detected silence 0.25s+ speech 0.06s: 5 clips
  Of which, no comma in the script (= likely extra sound): 4 clips

And the judgment values for those 4 clips were as follows:

base_64      Silence 1.96s+ speech 0.17s  Inserted 0   "気をつけて帰ってくださいね。"
conv_040     Silence 0.44s+ speech 0.27s  Inserted 0   "失礼いたします。"
conv_065     Silence 1.35s+ speech 0.16s  Inserted 0   "こちらで間違いございませんか。"
surprise_2   Silence 0.49s+ speech 1.09s  Inserted 0   "うそでしょう！？信じられません！"

All 4 had inserted 0. From the script matching perspective, they were perfect clips. Whether max_inserted was set to 3 or 0, these 4 clips would always pass.

Because STT does not pick up these sounds as characters. Sounds lasting 0.16-1.09 seconds after a silence are not recognized as meaningful words and do not appear in the transcription. As long as I was looking at the transcription, these clips appeared normal.

It's no wonder tightening the threshold didn't work; I wasn't detecting the target in the first place. I moved the threshold three times without seeing the target.

Solution: Change the Axis

The solution was not the threshold but adding another observation system. A waveform-based detector was incorporated into the generation pass judgment.

def trailing_artifact(wav_bytes, script_text,
                      gap_min=0.25, tail_min=0.06):
    """Detects "extra sounds" after finishing the script in waveform.

    ⚠️ Not detectable with Whisper matching. In actual measurement, 4 out of 198 corpus clips had
    speech of 0.16-1.09 seconds after silence of 0.44-1.96 seconds, all with inserted 0.
    ⚠️ Pauses for commas in the script are misdetected. Judgment is not made for scripts containing commas.
    """
    if "、" in (script_text or "") or "," in (script_text or ""):
        return None
    blocks = voiced_blocks(wav_bytes)
    if len(blocks) < 2:
        return None
    gap  = blocks[-1][0] - blocks[-2][1]
    tail = blocks[-1][1] - blocks[-1][0]
    return (gap, tail) if gap >= gap_min and tail >= tail_min else None

The threshold 0.25 / 0.06 was determined from the distribution of actually measured extra sounds (silence 0.26-1.96s, speech 0.07-1.09s). It wasn't started from a round number.

It was added in three places in the generation logic.

# ① Placed "before" pass judgment. Even if it passes, if detected in waveform, it is redrawn
art = verify.trailing_artifact(wav, text)
if art:
    logger.info(f"{line_key} steps={steps}: Extra sound NG"
                f"(Silence {art[0]:.2f}s+ speech {art[1]:.2f}s・Redraw)")
    continue

# ② In the final trimming, waveform silence positions are prioritized.
#    Whisper segment endpoints are unusable if extra sounds are not picked up as characters
end_sec = verify.artifact_free_end_sec(wav, text) or verify.script_end_sec(text, segs)

# ③ After trimming, confirm that no extra sounds remain before saving
if res2.ok and verify.trailing_artifact(trimmed, text) is None:
    save(trimmed)

And max_inserted was returned to 1. There was no basis for setting it to 0, and there was no reason to keep a setting that only reduced yield without effect.

Results

After retraining and measuring with the same 6 sentences:

Whisper match 6/6, no tail extension, 12 styles, sound pressure −16.6〜−17.9dB. All inspection items passed.

The clips rejected by the waveform gate completely matched the 4 identified in the preliminary scan.

4 times  base_64      "気をつけて帰ってくださいね。"
4 times  conv_040     "失礼いたします。"
4 times  conv_065     "こちらで間違いございませんか。"
4 times  surprise_2   "うそでしょう！？信じられません！"

"4 times" means "detected in all retries." Since this diffusion TTS is deterministic with the same caption and seed, the same extra sound is produced every time. There were no false positives, and the other 201 clips passed.

There were only 2 disqualifications (7 when max_inserted=0). Yield also recovered by returning the allowance to 1.

Pitfalls in Scanning

In the initial scan, all 12 were flagged because I included this in the probe sentences:

Let's begin.
  Main body → 0.40s silence → 0.75s speech   ← Judged as "extra sound"

It was a pause for a comma. There was a gap after "Let's," and "begin." continued, but the last block was part of the script. The condition "speech after the last gap" always results in false positives for sentences containing commas.

There are two countermeasures:

Limit probes to single sentences. Using only sentences without commas, speech after the main body can be definitively identified as extra. This was adopted.

Match with the script endpoint. Determine the script endpoint from Whisper segments and check if there is sound energy after that. More general but increases Whisper dependency.

Excluding false positives, the results were 29/72 → 16/72, as shown in the table above. If I had reported the initial results, it would have conveyed an incorrect sense of crisis: "All 12 failed."

Thresholds Only Work on Observed Targets

The lesson from this case is not "the threshold was too lenient." The threshold was moved three times without effect.

max_inserted=3 meant "allow up to 3 characters of noise," but also "learn up to 3 characters of hallucination." That much was correctly understood. So I tightened it.

What I overlooked was that the "inserted" counted by that threshold was only characters that appeared in STT transcription. Sounds not in the transcription were counted as inserted 0. I was trying to tighten a threshold that didn't include the target in its denominator.

This isn't limited to this case. When adjusting thresholds, two things should be confirmed:

How many will be lost if tightened. This was measured. 75 saved clips were re-judged, confirming no omissions with an allowance of 1, and 3 omissions with 0. The process itself was correct.

Whether the target is being observed with that threshold. This was never questioned. If asked once, I would have reached "there might be extra sounds in clips with inserted 0" and come up with the idea of looking at waveforms. In fact, scanning the material in waveform took 30 minutes. Moving the threshold three times and retraining took more than ten times that.

Another point is that the issue manifested in behavior, not data. Looking at the corpus data, "200 clips with inserted 0" doesn't seem abnormal. The quirk only appears after training and speaking. Dataset inspection alone is insufficient; the trained model must be tested with inputs not in the corpus.

And one more layer: the detector used for this article was created when I realized "STT doesn't find defects." Immediately after making it, I didn't notice that my quality gate relied only on STT. I had the knowledge but didn't apply it to my code. This pattern of the creator missing the hole occurred twice in this case.

Series: Mass-Producing Practical Voices from Diffusion TTS

This is a record of designing voices from a single caption, creating training corpora, and mass-producing practical voices for different roles. This article is Part 3: Quality Gates.

← Previous: The hallucination guard code only worked when there was no hallucination
→ Next: Rejecting candidates for fixable defects

All 18 Parts

1. The TTS chosen for sound quality was too slow for conversation
2. Voice Gacha
3. Having a machine select "narrator-like voices" from 24 candidates
4. The stricter the quality gate, the more flat takes survive
5. Speaking style is baked into the corpus
6. TTS that changes "recording location" every generation
7. One rough clip ruins the whole style
8. Where did the elongated ending come from?
9. The character that broke the TTS input
10. The hallucination guard code only worked when there was no hallucination 11. The "3 characters" allowed by the quality gate became a verbal tic ← You are here
11. Rejecting candidates for fixable defects
12. Defects not found in transcription
13. 70 minutes of training material lost in a network blink
14. From "ja" to "JP": Creating a babbling model
15. 4 registration paths, 0 management screens
16. Deployments kept overwriting each other's work
17. Measuring what isn't measured and tightening thresholds always fails

The insights are summarized in Mass-Producing Practical Voices from Diffusion TTS: Manufacturing Pipeline.