If you run a small service business or trade outfit, your phone is basically your cash register. The problem is, it’s also an open door for everyone trying to sell you SEO, commercial loans, or health insurance.
Even worse, these telemarketers use predictive auto-dialers that hit your phone 4 to 6 times in rapid succession. They spoof local prefixes so it looks like a neighbor or client is calling.
If you ignore the phone, you miss high-value homeowners and general contractors with immediate deadlines. If you answer, you end up wasting a few minutes with a call center while standing in the middle of a project.
Voicemail doesn't cut it, and simple number blocking doesn't work when they cycle through hundreds of virtual numbers.
So I built CallSmith AI: an autonomous inbound voice system that answers the phone, figures out who is calling in real time, and handles the situation accordingly.
The Strategy: Three Distinct Paths
Every incoming call falls into one of three buckets, and treating them all the same way is a recipe for lost money or wasted time:
-
The High-Value Homeowner: They want a quote for a remodel. They don’t want to leave a voicemail—they’ll just call the next person on Google. CallSmith talks to them, gets their rough scope, and immediately sends an SMS asking for site photos. If they text pictures back, the system flags them as high-priority and drops them into a
#vip-leadsDiscord channel. - The General Contractor: Commercial GCs don't need a sales pitch; they need paperwork. They want to know if you have active workers' comp, liability insurance, and a W-9. The agent explains our coverage limits, grabs the estimator's email, and automatically queues up our compliance packet (COI + W-9 + licenses) for delivery.
- The Persistent Telemarketer: Instead of hanging up, the agent plays along. Powered by Gemini, it uses dynamic stalling dialogue to tie up their phone lines. It formally delivers a Do Not Call (DNC) warning on the recorded line. If their auto-dialer calls back within that 4-to-6 burst window, it logs the transcript, timestamps, and caller ID into an immutable dossier for statutory TCPA claims ($500 to $1,500 per willful violation).
How the Stack Actually Works
Here’s the high-level architecture:
[ Inbound Call ]
│
â–¼
[ Twilio + STIR/SHAKEN ] ───(Flags Neighbor Spoofing)
│
â–¼
[ Cloud Run: FastAPI Microservice ]
│
├──► Gemini 3.5 Flash / Google GenAI SDK
├──► Google Cloud Text-to-Speech
│
â–¼
[ Deterministic Routing ]
├──► Homeowner ──► SMS Photo Request ──► #vip-leads (Discord)
├──► GC / Bids ──► Auto-Email Packet ──► #gc-compliance (Discord)
└──► Scammer ──► Stalling Loop ──► TCPA Evidence Locker
1. Telephony & The Anti-Spoofing Perimeter
Calls arrive via Twilio webhooks. Before doing anything heavy, the system checks Twilio’s STIR/SHAKEN attestation headers. If a call claims to be a local neighborhood number but has a "C" attestation (unverified gateway), it’s immediately flagged for closer scrutiny by the reasoning engine.
2. The Compute Layer: Google Cloud Run
The backbone is a small Python/FastAPI container deployed on Google Cloud Run. Telephony webhooks need near-instant responses to prevent dead air on the phone line. Cloud Run scales quickly from zero, handles the bidirectional audio stream, and coordinates calls to the Gemini API without running up idle server costs.
3. Real-time Reasoning with Gemini
We use Gemini via the new Google GenAI SDK for two critical jobs:
- Dynamic Stalling: Telemarketer scripts are designed to break out of basic IVR menus. Gemini dynamically produces context-aware responses to keep solicitors talking as long as possible.
- Structured Data Extraction: Once the conversation wraps, Gemini parses the messy conversational transcript into a strictly typed JSON schema.
What I Learned Building This
Audio Latency is Everything: On a phone call, a pause longer than 800 milliseconds makes people think the call dropped or realize they are talking to a machine. Keeping the Cloud Run containers warm and utilizing streaming responses from Cloud Text-to-Speech made a massive difference in conversational fluidity.
Turn Inconveniences into Data: Telemarketers count on people getting frustrated and slamming the phone down so they can just move to the next victim. Having an autonomous honeypot systematically log their script, company name, and call frequency completely flips the dynamic.
Keep the Logic Event-Driven: Connecting the backend directly to Discord channels via webhooks keeps the mobile experience clean. When a high-value homeowner texts back their project photos, seeing the images pop into #vip-leads with the transcript attached allows for an instant, informed follow-up.
Wrapping Up
Building this was a fun weekend challenge for the Google Cloud Hackathon. It solves a real daily headache by turning what used to be a distracting nuisance into an automated pipeline for genuine business leads and a structured defense against bad actors.
If you’re building voice agents or working on autonomous intake systems, I’d love to hear your thoughts on handling latency and anti-spoofing in the comments below!
