Don't Send Every Audio File Straight to the AI Model

An audio file can pass every technical checkpoint and still produce a bad AI result.

The upload completes. The decoder opens it. The model returns a response. The encoder writes a playable file. From the system's point of view, the job succeeded.

Then the user presses play and hears a voice that is still too quiet, missing pieces, or buried under artifacts.

It is tempting to blame the model immediately. Sometimes that is fair. But many bad jobs begin earlier, when the product treats every decodable file as a valid model input.

A recording can be almost silent, heavily clipped, mostly empty, unexpectedly multichannel, or far outside the conditions a model was trained to handle. Sending all of those through the same pipeline with the same settings turns predictable input problems into mysterious model failures.

I prefer to put a small policy layer in front of inference:

upload -> probe -> inspect -> route -> process -> validate -> preview

The goal is not to diagnose audio perfectly. It is to catch obvious problems, choose a safer path when possible, and avoid claiming success when the result cannot be trusted.

A container is not the signal

Start with the inexpensive facts. Does the file contain an audio stream? What codec, duration, sample rate, channel count, and channel layout did the decoder find?

ffprobe can return that metadata as JSON without decoding the whole signal:

ffprobe -v error \
  -select_streams a:0 \
  -show_entries stream=codec_name,sample_rate,channels,channel_layout:format=duration \
  -of json \
  input.mp4

The ffprobe documentation covers -show_entries, stream selection, and its JSON writer in more detail.

These checks catch more than malformed uploads. They also stop quiet assumptions from spreading through the pipeline. A speech model expecting mono audio should not discover a six-channel layout halfway through inference. Blindly downmixing is not always harmless either: channels can contain different microphones, or partially cancel when combined.

Metadata still cannot tell us whether the file contains useful audio. For that, inspect at least a bounded portion of the decoded signal. Useful first-pass measurements include:

• peak and RMS level
• long silent regions
• samples at or near full scale
• active audio or speech ratio
• channel imbalance

FFmpeg's astats and silencedetect filters are enough for a basic server-side pass:

ffmpeg -hide_banner -i input.mp4 \
  -af "astats=metadata=1:reset=0,silencedetect=n=-50dB:d=1" \
  -f null -

The silence level and duration above are examples, not standards. A whispered interview, a screen recording, and a field recording should not share thresholds just because they are all audio files. For speech products, a voice activity detector is also more useful than treating every non-silent sound as speech.

Low level is a routing decision

One especially awkward case is a recording whose digital level is extremely low.

The obvious fix is to normalize it before inference. That can help place the signal inside a model's expected operating range, but it does not create information that was never captured. Raising gain lifts the voice and the noise floor together. It does not improve signal-to-noise ratio, repair a poor microphone position, or restore detail lost during recording.

This is why I would not normalize every upload to the same target. First decide whether the input is unusually low for the pipeline. If it is, apply bounded gain with headroom, then run the model. If the recording is already clipped or contains almost no usable speech, gain is the wrong intervention.

While working on CleanAudio, this distinction became important: very low-level inputs may need a calibration step, but that is input conditioning, not a promise that every result should have the same loudness.

The thresholds belong to the model and the use case. They should come from tested failures, not from numbers copied out of a mastering guide.

Turn measurements into a plan

I find it more useful to produce a processing plan than a single valid boolean. A file can be valid enough to decode but still deserve conservative processing or manual review.

Here is a simplified policy in TypeScript:

interface AudioInspection {
  decodable: boolean;
  audioStreamCount: number;
  durationSeconds: number;
  peakDbfs: number;
  rmsDbfs: number;
  clippedSampleRatio: number;
  activeAudioRatio: number;
}

interface AudioPolicy {
  minDurationSeconds: number;
  maxDurationSeconds: number;
  minActiveAudioRatio: number;
  maxClippedSampleRatio: number;
  calibrateBelowDbfs: number;
  targetInputRmsDbfs: number;
  maxCalibrationGainDb: number;
  maxAllowedPeakDbfs: number;
}

interface ProcessingPlan {
  disposition: "process" | "review" | "reject";
  mode: "standard" | "conservative";
  preGainDb: number;
  reasons: string[];
}

function planAudio(
  input: AudioInspection,
  policy: AudioPolicy,
): ProcessingPlan {
  const reasons: string[] = [];

  if (!input.decodable || input.audioStreamCount === 0) {
    return {
      disposition: "reject",
      mode: "standard",
      preGainDb: 0,
      reasons: ["No decodable audio stream"],
    };
  }

  if (
    input.durationSeconds < policy.minDurationSeconds ||
    input.durationSeconds > policy.maxDurationSeconds
  ) {
    return {
      disposition: "reject",
      mode: "standard",
      preGainDb: 0,
      reasons: ["Duration is outside the supported range"],
    };
  }

  if (input.activeAudioRatio < policy.minActiveAudioRatio) {
    return {
      disposition: "review",
      mode: "conservative",
      preGainDb: 0,
      reasons: ["Too little active audio was detected"],
    };
  }

  let mode: ProcessingPlan["mode"] = "standard";

  if (input.clippedSampleRatio > policy.maxClippedSampleRatio) {
    mode = "conservative";
    reasons.push("Input contains substantial clipping");
  }

  let preGainDb = 0;

  if (input.rmsDbfs < policy.calibrateBelowDbfs) {
    const gainTowardTarget = policy.targetInputRmsDbfs - input.rmsDbfs;
    const availableHeadroom = policy.maxAllowedPeakDbfs - input.peakDbfs;

    preGainDb = Math.max(
      0,
      Math.min(
        gainTowardTarget,
        availableHeadroom,
        policy.maxCalibrationGainDb,
      ),
    );

    if (preGainDb > 0) reasons.push("Low-level input needs bounded gain");
  }

  return {
    disposition: "process",
    mode,
    preGainDb,
    reasons,
  };
}

There are deliberately no magic values in this example. A threshold is part of the product policy, not a universal property of audio. Version it alongside the model, record which route each job takes, and review the files clustered near a boundary.

This policy layer also makes failures easier to explain. "We could not detect enough usable audio" is more actionable than "processing failed." A conservative route can preserve more of the original signal instead of applying the strongest available effect to a risky input.

The output needs a postflight check

A successful inference request only proves that inference ran. Before offering the result, decode it again and check the boring invariants:

• expected duration and timeline were preserved
• output is not silent or unexpectedly clipped
• sample rate and channel mapping are supported
• the encoder produced a complete, playable file
• any known model or codec delay was compensated

Compare those measurements with the input, not only with fixed limits. A large, unexplained duration change is suspicious even when both files are individually valid.

These checks catch catastrophic failures. They cannot tell whether consonants were softened, room tone started pumping, or a voice became less natural. That still requires listening. In a previous article, I described a synchronized A/B preview for making that comparison without restarting two separate players.

The distinction matters: automated validation protects the pipeline, while the preview protects the user's judgment.

Let the product admit uncertainty

AI interfaces often compress several states into one green message: Enhancement complete.

But there is a meaningful difference between a model completing, a file passing structural checks, and a person deciding that the result is better. Treating them as the same event makes the product sound more certain than the system actually is.

The original file should remain available. Risky inputs should get a clear explanation. A result that passes only basic checks should still be presented as something to review, not as an unquestionable improvement.

The model is only one stage in the feature. A small amount of inspection before and after it can prevent predictable failures, make routing decisions visible, and give the user a more honest result.

Sometimes the best AI processing decision is to do less. Sometimes it is not to run the model at all.

Disclosure: I am involved in building CleanAudio, an AI audio and video noise-removal tool. The implementation pattern and opinions in this article are presented as general product and engineering guidance.

AI assistance disclosure: AI tools assisted with drafting and editing this article. The technical direction, product observations, and final review are the author's.