Safe Voice Commands for My Home Automation Server

Introduction

Running a home‑automation server from my garage in Pompano Beach has been a hobby that turned into a daily workhorse for Dream Futures Tech. I wanted to control lights, the coffee maker, and the garage door with my voice, but I also needed to make sure that a stray phrase or a mis‑heard command didn’t trigger something dangerous. The solution I landed on is simple: filter everything through a whitelist regular expression before handing the text to a language model. In this article I walk through the exact steps I took, the code I wrote, and the pitfalls I avoided.

The Problem

Voice assistants are great, but they are also black boxes. When you feed raw speech‑to‑text output into a large language model (LLM) you hand it free reign to interpret anything it sees. A mis‑recognition like “turn off the oven” becoming “turn off the oven door” could be harmless, but “unlock the front door” misheard as “unlock the front store” could expose a business. I needed a deterministic gate that only allowed commands I explicitly approved.

Why Regex First?

A regular expression is deterministic, fast, and easy to audit. By defining a whitelist of patterns that represent the actions I actually want to perform, I can reject everything else outright. This gives me three benefits:

1. Safety – No surprise commands slip through.
2. Performance – The regex check runs in a few microseconds, far cheaper than an API call.
3. Transparency – The list of allowed phrases lives in source control, so any change is tracked.

Building the Whitelist

I kept the whitelist in a JSON file so I could edit it without touching code. Each entry contains a pattern and a short description.

[
  {"pattern": "^turn on (the )?lights?$", "action": "lights_on"},
  {"pattern": "^turn off (the )?lights?$", "action": "lights_off"},
  {"pattern": "^open (the )?garage door$", "action": "garage_open"},
  {"pattern": "^close (the )?garage door$", "action": "garage_close"},
  {"pattern": "^brew coffee$", "action": "coffee_start"}
]

The patterns are deliberately simple – I avoid optional words and wildcards that could broaden the match unintentionally. When the server receives a transcription from the microphone, it iterates over the list and picks the first match.

Integrating the LLM

Even with a whitelist, I still want the LLM to handle variations like “please turn the lights on” or “could you open the garage door?”. The trick is to run the regex first, and only if it fails do I fall back to the LLM with a very narrow prompt that asks for a command only from the allowed set.

const fetch = require('node-fetch');
const whitelist = require('./whitelist.json');

function matchWhitelist(text) {
  for (const entry of whitelist) {
    const re = new RegExp(entry.pattern, 'i');
    if (re.test(text)) return entry.action;
  }
  return null;
}

async function askLLM(text) {
  const prompt = `You are a voice‑command parser for a home automation system. Only respond with one of the following keywords: ${whitelist.map(e=>e.action).join(', ')}. If the request does not match any, reply with "unknown".
User: ${text}`;
  const response = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({model: 'gpt-4o-mini', messages: [{role: 'user', content: prompt}], temperature: 0})
  });
  const data = await response.json();
  return data.choices[0].message.content.trim();
}

async function handleCommand(transcript) {
  let action = matchWhitelist(transcript);
  if (!action) {
    action = await askLLM(transcript);
  }
  if (action && action !== 'unknown') {
    // dispatch to the hardware controller
    console.log('Executing:', action);
    // e.g., send MQTT message or call a local script
  } else {
    console.warn('Command rejected:', transcript);
  }
}

Key points:

• Temperature 0 forces deterministic output from the LLM.
• The prompt explicitly limits the LLM’s vocabulary, reducing hallucination risk.
• If the LLM still returns "unknown", the command is dropped.

Putting It All Together

My server runs on a Raspberry Pi 4 with Node.js 20. I use the speech-to-text library from the local microphone to get a raw string, then feed it into handleCommand. The flow looks like this:

1. Capture audio → speech‑to‑text.
2. Run matchWhitelist. If a match, execute immediately.
3. If no match, call askLLM with the constrained prompt.
4. Execute the returned action or log a rejection.

Because the whitelist check happens first, most everyday commands (lights, garage) are processed locally without any network latency. Only the occasional polite phrasing hits the LLM, and even then the response is bounded.

Lessons Learned

• Keep the whitelist tight. Adding a new pattern should be a deliberate code change, not a casual addition.
• Log rejections. I store every rejected transcript in a SQLite table; reviewing them helped me spot common mis‑recognitions and expand the whitelist safely.
• Rate‑limit the LLM. I set a per‑minute cap (10 calls) to avoid surprise bills if the microphone picks up background chatter.
• Test with real speech. Synthetic text works, but the acoustic environment in my garage introduced quirks that only field testing revealed.

Conclusion

By front‑loading a deterministic regex whitelist and only falling back to a language model when needed, I got a voice‑controlled automation system that feels responsive and, more importantly, safe. The pattern is portable: any small business that wants voice control—whether it’s a local bakery turning on the oven or a plumbing shop opening a service door—can adopt the same two‑layer approach.

I build sites and automations for small businesses at dreamfuturestech.com.