Build a Voice Assistant with Python and Whisper

tags: python, ai, audio, tutorial

Imagine hearing your computer respond to your voice with the same nuance and clarity as a human, all running locally on your own machine without sending a single byte of data to the cloud. That’s the power of building a Voice Assistant with Python and OpenAI’s Whisper model. While cloud-based assistants like Siri or Alexa are convenient, they come with privacy trade-offs and latency issues. By combining Whisper’s state-of-the-art speech-to-text capabilities with Python’s flexibility, you can create a private, fast, and customizable voice interface that answers your questions, controls your scripts, or even just chats with you—right from your terminal.

Let’s get your hands dirty and build one from scratch today.

Why Whisper and Python?

OpenAI’s Whisper is a transformer-based model trained on 680,000 hours of multilingual and multitask supervised data. Unlike older speech recognition tools that struggled with background noise or specific accents, Whisper delivers high accuracy across diverse environments [5]. It’s open-source, meaning you can run it locally, ensuring your conversations stay private.

Python is the natural partner for this task. With libraries like SpeechRecognition, pyttsx3, and openai-whisper (or its faster variant faster-whisper), you can stitch together the entire pipeline: capturing audio, transcribing it, processing the text with an LLM, and speaking the response back [6][7].

Setting Up Your Environment

Before writing code, you need the right tools. Start by creating a dedicated project directory and a virtual environment to keep dependencies clean.

mkdir voice-assistant && cd voice-assistant
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Next, install the core speech processing stack. While openai-whisper works well, faster-whisper is often preferred for production due to its optimized performance and lower memory usage [2][5].

pip install SpeechRecognition==3.10.4 \
faster-whisper \
pyttsx3==2.98 \
python-dotenv==1.0.1

You’ll also need an API key if you plan to send the transcribed text to an LLM like OpenAI’s GPT or Anthropic’s Claude. Store this securely in a .env file to avoid hardcoding secrets:

# .env
ANTHROPIC_API_KEY=sk-ant-api03-your-key-here
OPENAI_API_KEY=your-openai-key-here

The Core Logic: Speech-to-Text and Text-to-Speech

The heart of your assistant is the loop: Listen → Transcribe → Process → Speak. Let’s break down the first two steps.

Capturing and Transcribing Audio

We’ll use SpeechRecognition to capture microphone input and faster-whisper to transcribe it. The recognizer object handles the audio stream, while the whisper model converts audio waves into text.

import speech_recognition as sr
from faster_whisper import Whisper
import os
from dotenv import load_dotenv

# Load API keys
load_dotenv()

# Initialize Whisper model (small.en is fast and accurate for English)
model = Whisper("small.en")

# Initialize Speech Recognition
recognizer = sr.Recognizer()

def listen_for_prompt():
    with sr.Microphone() as source:
        print("Listening... (Say something)")
        recognizer.adjust_for_ambient_noise(source, duration=0.5)
        audio = recognizer.listen(source, timeout=5)

    # Convert audio to WAV file for Whisper
    audio_path = "prompt.wav"
    with open(audio_path, "wb") as f:
        f.write(audio.get_wav_data())

    # Transcribe with Whisper
    segments, _ = model.transcribe(audio_path)
    text = "".join([segment.text for segment in segments])

    print(f"You said: {text}")
    return text

This function waits for you to speak, saves the audio to a temporary file, and runs it through Whisper. The small.en model is a great starting point—it’s lightweight (about 50MB) and transcribes 5 seconds of audio in under 0.5 seconds on modern hardware [6].

Speaking the Response Back

Once you have text, you need to turn it into speech. pyttsx3 is a simple, offline text-to-speech engine that works without external APIs.

import pyttsx3

engine = pyttsx3.init()

def speak(text):
    print(f"Assistant: {text}")
    engine.say(text)
    engine.runAndWait()

Putting It All Together: The Assistant Loop

Now, let’s combine transcription and speech into a working loop. For this example, we’ll use a simple hardcoded response. In a real project, you’d send the transcribed text to an LLM API for dynamic answers [3][7].

def main():
    speak("Hello! I'm your local voice assistant. What can I do for you?")

    while True:
        user_input = listen_for_prompt()

        if not user_input:
            continue

        # Simple logic: if user says "stop", exit
        if "stop" in user_input.lower():
            speak("Goodbye!")
            break

        # Placeholder for LLM integration
        response = f"I heard you say: '{user_input}'. How can I help with that?"
        speak(response)

if __name__ == "__main__":
    main()

Run this script, and you’ll have a working voice assistant that listens, transcribes, and responds. The beauty here is that everything runs locally—no cloud dependencies for the speech parts, and you can swap the LLM logic to use any provider you prefer.

Enhancing Your Assistant

Once the basics are working, you can level up with these practical additions:

Wake Word Detection: Instead of listening constantly, detect a specific phrase like “Hey Assistant” to activate the loop [2]. This saves CPU and reduces false triggers.
Context Management: Pass previous conversations to your LLM so the assistant remembers context. This creates a more natural, conversational experience [3].
Custom Commands: Map specific phrases to system actions (e.g., “Open browser” → subprocess.run(["google-chrome"])).
Faster Models: If you need speed, try tiny.en or base.en models. They’re less accurate but faster for real-time applications [6].

Privacy and Performance Considerations

Running Whisper locally means your audio never leaves your machine, a critical feature for privacy-conscious users [5]. However, be aware that larger models (like large-v3) require significant RAM and GPU resources. For most desktop use cases, small.en or base.en offers the best balance of speed and accuracy.

If you’re on a low-power device, consider using faster-whisper with quantized models, which can reduce memory usage by up to 50% while maintaining accuracy [2].

Start Building Today

You now have a fully functional, private voice assistant built with Python and Whisper. The code is simple, the dependencies are open-source, and the results are immediate. Don’t just stop at transcription—integrate an LLM, add wake word detection, or connect it to your home automation system.

The best part? You can do all of this today without waiting for a cloud service to approve your API key. Clone the logic, tweak the models, and make it yours.

Ready to go further? Try adding a wake word or integrating Anthropic’s Claude for smarter responses. Share your project on Dev.to and let the community see what you built. The future of voice interaction is private, fast, and entirely in your hands.

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.