What Happens When Assigning Frequency Bands to 5 Vowels? - Misclassification and Spectral Tilt Verified in Synthetic Speech of 3 Speakers

📝 Originally published (in Japanese) at forge.workstyle.tech.

Spectral tilt refers to the tendency of audio levels to decrease from low to high frequencies, for example.

Target Audience: Developers who want to analyze audio using the Web Audio API and animate VRM lip shapes based on vowels.

Input and Output: Takes the frequency spectrum of synthesized speech as input and outputs candidate vowels (aa / ih / ou / ee / oh).

What You'll Gain from This Article: Understanding why directly mapping frequency bands to vowels failed and how to improve feature extraction.

Assigning Vowel Names to Frequency Bands

I wanted a browser-based VRM avatar to select lip shapes based on vowels in response to audio. My initial approach was to divide the frequency axis into bands and use the energy in each band as a score for a specific vowel. The visemes in the table below represent lip shapes, with aa corresponding to the Japanese vowel "a," for example.

By passing these scores through a softmax function, I could treat them as weights for the lip shapes. At first glance, this seemed like a straightforward design, utilizing different frequency bands from low to high.

However, simply labeling a band as aa doesn’t guarantee it represents the vowel "a." Softmax only converts scores into weights; it doesn’t learn the mapping between bands and vowels. If the score relationships are incorrect, it will incorrectly emphasize the wrong candidates.

To verify this, I used labeled sustained vowels (e.g., prolonged "a" sounds) synthesized by our in-house TTS system, covering 3 speakers × 5 vowels. The observations below are specific to this evaluation and do not represent general vowel frequency characteristics.

Most Assignments Responded to the Wrong Vowels

Investigating which vowels actually had the highest energy in each band revealed discrepancies from the initial assumptions.

Only ee matched the expected vowel.

In the deviation table, "a" showed significant increases in the mid-range: +16.4dB at 900–1100Hz, +16.6dB at 1100–1400Hz, and +15.8dB at 1400–1700Hz. These deviations indicate how much higher the energy was compared to a baseline. The old implementation primarily classified these mid-range increases as oh or ou.

This doesn’t mean "a" lacks low-frequency energy. The issue was that the energy peak used to distinguish "a" didn’t align with the band assigned to aa.

Comparing Absolute Levels Also Captures Spectral Tilt

In addition to the incorrect mapping, comparing absolute band levels introduced another problem.

When bands compete directly based on their values, not only the local shapes distinguishing vowels but also the spectral tilt influence the scores. In the old implementation, aa, assigned to the low-frequency band, was often selected due to this tilt.

However, I’m not generalizing that low frequencies are always the highest in all audio. Results vary depending on band width, aggregation methods, speakers, and audio sources. The issue here was that, with the given input and implementation, absolute level comparisons failed to distinguish vowels effectively.

Initially, I thought averaging in dB was the main cause. Unlike linear power averages, dB averages are influenced by weaker bins. While this was theoretically an issue, switching to linear power averaging worsened the bias toward aa.

Correcting the aggregation method doesn’t guarantee suitable features for classification. The main cause here wasn’t just the averaging unit.

Verifying Values Extracted via Web Audio API

The core of calculating band values is as follows. The analyser is a dedicated node connected to the playback stream, not the destination.

const bands = [
  ["aa", 250, 700],
  ["oh", 700, 1200],
  ["ou", 1200, 2000],
  ["ee", 2000, 3000],
  ["ih", 3000, 5000],
];

const spectrum = new Float32Array(analyser.frequencyBinCount);

function readBandLevels() {
  analyser.getFloatFrequencyData(spectrum); // dB values

  return bands.map(([viseme, low, high]) => ({
    viseme,
    level: meanBandPowerDb(
      spectrum,
      analyser.context.sampleRate,
      analyser.fftSize,
      low,
      high
    ),
  }));
}

meanBandPowerDb is a helper function that selects bins within a band, converts dB to linear power, averages them, and converts back to dB. It uses the actual sample rate and FFT size for frequency-to-bin mapping.

This code outlines the failed approach. Using the returned values directly as vowel weights retains both the mapping errors and the influence of spectral tilt. It’s essential to separately verify that values are obtained from the API and that vowels are correctly identified.

Switching to Deviations from Long-Term Averages

The improvement involved focusing on how much each band deviated from its long-term average, rather than absolute levels. For example, if the running average from past audio is −30dB and the current level is −20dB, the difference of +10dB is used.

function subtractBaseline(levels, baseline) {
  return levels.map((level, i) => level - baseline[i]);
}

This is preceded by updating the running average for each band. A running average incorporates new frame values incrementally, capturing relatively stable components like spectral tilt and speaker timbre in the baseline. Deviations then represent changes in the current sound.

However, tilt and timbre aren’t always constant. Sudden input level changes persist as deviations until the average catches up. If speaker changes are possible (e.g., during reconnections), the design should not carry over averages.

Additionally, sustaining the same vowel causes the average to shift toward that vowel, reducing the deviations used for distinction. This method assumes vowel transitions within the input. Sustained vowels serve as a stress test for this weakness.

Large Bands Alone Can’t Capture Decreases

Does simply selecting the band with the largest deviation suffice? Not quite.

In the table, "i" showed −16.4dB at 1100–1400Hz and −16.0dB at 1400–1700Hz, but +11.9dB at 2600–3200Hz. "I" doesn’t lack positive features; it exhibits a combination of increases and decreases.

Similarly, "u" showed −18.5dB at 2600–3200Hz and −15.4dB at 3200–4000Hz. Simply linking large values to vowels fails to adequately handle these negative features.

The new implementation thus moved toward matching multi-band deviation patterns with vowel templates. For example, a template for "i" might specify mid-range decreases and high-range increases. Additionally, centering by removing uniform changes across all bands is necessary. For instance, if three bands show +2, +4, and +6dB, subtracting the average (+4dB) yields −2, 0, and +2dB. This prevents overall volume changes from being mistaken for vowel-specific shapes.

Accuracy Requires Contextual Measurement

Comparisons between the old and new implementations were made within the same evaluation harness. The harness inputs vowel sequences, compares correct labels with estimates, and aggregates results.

The old implementation fell below the uniform random expectation of 20% in all cases. This indicates not just random errors but systematic misclassifications in the band assignments.

Meanwhile, 71.3% is not real-speech accuracy. It was achieved by segmenting sustained vowels into 120ms chunks, randomly ordering them, and evaluating short transitions to simulate real-speech speed. Continuous speech with natural consonant-vowel transitions wasn’t tested. Additionally, the new implementation includes multiple changes, so this improvement can’t be attributed solely to introducing long-term averages.

Before assigning vowel names to bands, measure what each band actually responds to. In this case, the necessary step was moving from absolute level competition to comparing spectral shapes, including increases and decreases, before fine-tuning boundaries.