Building an AI Voice Transcription Feature — Lessons That...
Active Document
The 2024 dev journey behind shipping a Whisper-powered voice transcription feature on Letterbird — and what those lessons taught us about running 24/7 AI voice agents for Edmonton businesses in 2026.
Loading document...
In September 2024, I shipped a voice transcription feature into Letterbird — a quick-note app where users record audio and have it transcribed in real time via OpenAI's Whisper API. I filmed the whole build as a day-in-the-life so other developers could see the messy middle: the debugging, the wrong turns, the "why is this API route returning 401?" moments.
Eighteen months later, voice AI is no longer a feature you bolt onto a side project. At Agency7, we now ship full AI voice agents for Edmonton businesses — agents that answer calls, book appointments, qualify leads, and hand off to a human when they should. The Letterbird transcription project turned out to be the foundation. This post revisits what I learned building it and maps those lessons onto what voice AI actually costs, breaks, and delivers in 2026.
👉 Watch the original 2024 build video
The project had a simple premise: press a button, record audio in the browser (or on iOS), POST the file to a Next.js API route, call Whisper, return the transcript. Three moving parts, each with its own failure mode.
audio/webm on desktop and audio/mp4 on Safari.whisper-1 endpoint and returned the text.The painful parts weren't the AI. They were the seams between pieces: iOS Safari silently dropping audio when the tab lost focus, Next.js file-size limits tripping on longer recordings, and WebM files that played fine in the browser but that Whisper couldn't parse because the duration metadata was missing. Each of those took a day.
Got right:
Would change:
Transcription is the batch version of voice AI: record, send, wait, display. A voice agent is the streaming version: the caller is still speaking, and the agent needs to understand, reason, and respond in under 700 milliseconds or the conversation feels dead. That's the whole game in 2026.
The Letterbird transcription work taught me the substrate — file formats, codecs, API error handling, retry logic, cost-per-minute math — but voice agents add:
Every one of those is a rabbit hole. The 2024 "record and transcribe" flow is now roughly 5% of a production voice-agent build.
Rough numbers from Agency7 engagements with Edmonton service businesses:
If you want the longer breakdown, we wrote it up here: AI Voice Agents for Edmonton Trades vs an Answering Service.
audio/webm;codecs=opus vs. audio/mp4 is the single most common source of silent bugs in browser audio.For asynchronous transcription of recorded audio in English, Whisper (v3 and its hosted variants) is still the default cost-accuracy pick. For live streaming voice agents, Deepgram Nova-3 and AssemblyAI Universal-2 typically beat Whisper on latency and real-time diarization.
Yes, but you usually shouldn't. Running Whisper locally requires GPU infrastructure, scaling logic, and monitoring. For under $0.006/minute you can hit OpenAI's hosted endpoint and skip the entire ops burden. Self-hosting only makes sense when you have strict data-residency requirements (e.g., healthcare records that can't leave Canada).
Transcription is one-way: audio goes in, text comes out. A voice agent is a two-way conversation — it listens, thinks, speaks, remembers context across turns, and takes actions (booking appointments, looking up information, routing calls). Transcription is a component of a voice agent, not the whole thing.
End-to-end (caller finishes speaking → agent starts responding) should be under 800ms. Under 500ms feels snappy. Over 1.2 seconds feels robotic and callers start repeating themselves. Hitting under 800ms consistently requires streaming ASR, a fast LLM like GPT-4o-mini or Claude Haiku 4.5, and a low-latency TTS like ElevenLabs Turbo.
Yes. Whisper, Deepgram, and ElevenLabs all handle English + Spanish natively. For Edmonton's Cantonese, Mandarin, Punjabi, and Tagalog-speaking customer bases, model quality is good but not as polished — we typically pair a multilingual ASR with an English-first LLM and explicit language-detection routing.
A focused build for a single workflow (e.g., dental appointment booking, HVAC after-hours triage) is typically 2-4 weeks. Multi-workflow agents with deep CRM integration (HubSpot, ServiceTitan, Jobber) take 4-8 weeks. The long pole is almost always the CRM API integration, not the AI.
Letterbird's transcription feature still works. The Whisper call pattern I shipped 18 months ago is the same pattern we use for every dashboard-style transcription tool. What's changed is the ambition — we're no longer recording audio and waiting. We're having full conversations with callers, on behalf of Edmonton businesses, at 3 a.m.
If you're an Edmonton business owner wondering whether voice AI is ready for your operation, the answer is usually yes — but the framing matters. Don't ask "can it transcribe?" Ask "can it take the call I'm losing at 6:37 p.m. on a Tuesday?" That's the question we build for.
Curious what your business's voice agent would look like? Book a free conversation at consult.agency7.ca/ai-audit or see what we offer on the AI Voice Agents page.