How to Run Whisper Locally for Real-Time Speech-to-Text (faster-whisper)

Run Whisper locally for real-time speech-to-text

For live translation and voice apps, you want speech-to-text that's fast and free per use — which
means running Whisper locally with faster-whisper. It pairs perfectly with APIVAI for the
translation step: Whisper turns audio into text locally, APIVAI (GPT-5.5) translates it cheaply over
an OpenAI-compatible call, and you keep latency low and per-second cost at zero for the audio part.

This guide gets local Whisper running for real-time use.

Install

pip install faster-whisper

faster-whisper is a fast reimplementation of OpenAI's Whisper using CTranslate2. It runs on GPU
(CUDA) or CPU.

Choose a model size

For live captioning, small on a GPU is the sweet spot.

Transcribe in near real time

from faster_whisper import WhisperModel

model = WhisperModel("small", device="cuda", compute_type="float16")  # device="cpu", compute_type="int8" for CPU

def transcribe(audio_path: str, lang: str = "zh") -> str:
    segments, _ = model.transcribe(audio_path, language=lang, vad_filter=True, beam_size=1)
    return "".join(s.text for s in segments).strip()

• Capture short audio chunks (1–3s) from the mic/OBS monitor and feed them in.
• vad_filter=True skips silence so you don't transcribe noise.
• beam_size=1 favors speed for live use.

Hook it to translation (APIVAI)

from openai import OpenAI
ai = OpenAI(api_key="YOUR_APIVAI_API_KEY", base_url="https://api.apivai.com/v1")

def translate(text, target="Spanish"):
    r = ai.chat.completions.create(model="gpt-5.5", messages=[
        {"role": "system", "content": f"Translate to natural {target}. Output only the translation."},
        {"role": "user", "content": text}])
    return r.choices[0].message.content.strip()

print(translate(transcribe("chunk.wav")))

Performance tips

• GPU (float16) is much faster than CPU; on CPU use int8 and a smaller model.
• Pre-load the model once and reuse it; don't reload per chunk.
• Keep chunks short; longer audio adds latency.

FAQ

Is local Whisper free? Yes — faster-whisper runs on your own hardware; there's no per-call
cost. You only pay for the translation step (e.g. GPT-5.5 via APIVAI), which is cheap.

Does APIVAI provide Whisper? No — APIVAI provides the OpenAI/Anthropic-compatible chat models
(the translation step). Run Whisper locally for speech-to-text.

Which model size for live captions? small on a GPU — a good balance of speed and accuracy.

What latency can I expect? With small + short chunks, transcription is sub-second; end-to-end
(STT → translate → caption) is typically ~1–2s.

Get started

Install faster-whisper, run the transcribe loop, and send the text to APIVAI for translation.
Examples: APIVAI examples repo.