--- title: 'Speech-to-text pipelines that survive real audio' excerpt: 'Pavleur streams live meeting audio through Deepgram and ElevenLabs. Transcription turned out to be the easy part. The pipeline is mostly failure handling.' date: '2026-06-22' lastModified: '2026-07-04' author: 'Teo Deleanu' authorAvatar: '/team/teo.jpg' tags: ['Production AI', 'AI Engineering', 'Speech-to-text', 'Realtime'] keywords: ['speech-to-text pipeline', 'realtime transcription architecture', 'Deepgram vs ElevenLabs', 'streaming audio websocket'] featured: false tldr: 'Pavleur, my own live work assistant, streams meeting audio to Deepgram Nova-3 and ElevenLabs Scribe v2 through one proxy, and the STT integration was days of work while the failure handling took months. Real audio broke the clean design three separate ways: a provider that idle-closes after 16 seconds of the silence you deliberately stopped sending, a browser cap of roughly 6 AudioContexts that killed every seventh session, and network blips that eat audio you can never re-request. The durable answer was a keepalive of silent PCM, one AudioContext for the process lifetime, a 2.2MB replay buffer, and churn telemetry that swaps providers automatically. Live audio is the one input you cannot retry, so the pipeline has to be built around never dropping it.' keyTakeaways: - 'Live audio is unreplayable input. A document pipeline retries a failed stage; a meeting will not repeat itself. Buffer through every reconnect or lose the words forever.' - 'Your bandwidth optimization is the provider''s liveness signal: filtering silence client-side triggered upstream idle-closes every 16 seconds, fixed by a silent-PCM keepalive that VAD ignores.' - 'Platform ceilings hide in the capture layer. Chromium caps AudioContexts around 6 per page, so the fix was one context for the process lifetime, suspended instead of closed.' - 'Route providers by what each does best and instrument for churn: reconnect counts, longest transcript gap, and an automatic provider swap when one upstream degrades.' --- Pavleur, the live work assistant I build and run, streams meeting audio into a transcript in real time and feeds that transcript to a model. Getting a speech-to-text provider to return words took a few days. Keeping a transcript alive through a real hour-long meeting, on real networks, across two providers with different failure moods, is where the engineering went. This post is that pipeline, with first-party receipts: it is my own product, live at [pavleur.com](https://pavleur.com), so nothing here hides behind an NDA. The honest summary up front: transcription is the easy third of a transcription product. The rest is failure handling, and real audio finds failures a document pipeline never meets, because live audio is the one input you cannot ask for again. ![Pavleur speech-to-text pipeline: Electron capture at 16kHz mono in 128ms chunks, a Fly.io proxy with silent-PCM keepalive and a 2.2MB reconnect buffer, routed to Deepgram Nova-3 or ElevenLabs Scribe v2 by language, with churn telemetry and automatic provider swap underneath](/blog/stt-pipeline-anatomy.svg) ## The clean pipeline, as designed On the whiteboard the system is tidy. An Electron client captures the microphone and, for meetings, the system audio too. Audio is resampled to 16kHz, mixed to mono, and converted to 16-bit PCM, then shipped in 2,048-sample chunks, about 128 milliseconds each, over a WebSocket to a proxy on Fly.io. The proxy exists because API keys, quotas, and plan enforcement do not belong in a desktop binary. It also owns the routing decision: English sessions go to Deepgram Nova-3, which gets 50 concurrent slots; other languages go to ElevenLabs Scribe v2, which gets 6. Both stream interim results back, so words appear while the speaker is mid-sentence. Commit boundaries come from voice activity detection: a segment finalizes after one second of silence, with 200 milliseconds minimum speech and 500 milliseconds minimum silence so breaths and clicks do not become utterances. That design is real and it ships. It is also maybe a third of the code that actually runs. The rest accumulated one production surprise at a time. ## The provider that hangs up on silence To save bandwidth, the client filters silence before it ever reaches the wire. Meetings are full of pauses, so most of the time nothing is being sent, which is exactly the optimization you would make. ElevenLabs' realtime endpoint idle-closes the upstream connection after roughly 16 seconds without an audio chunk. Put those two sensible decisions together and the product does something absurd: every time the room goes quiet for a stretch, the provider hangs up, the proxy reconnects, and the user watches a "reconnecting" indicator flicker in the middle of a perfectly healthy session. The fix is a keepalive that sends frames of silent PCM every 8 seconds during pauses. Zeroed samples never trip VAD, so they cannot corrupt the transcript, but they reset the provider's idle clock. It works, and it is load-bearing, and it is the kind of fix nobody diagrams: my bandwidth optimization was the provider's liveness signal, and neither side was wrong. ## The browser resource that ran out The capture layer died in a way that took real effort to see. Chromium hard-caps AudioContexts at roughly 6 per page, and closing one does not free the slot immediately, because the release rides on garbage collection. Pavleur created a fresh AudioContext per session and closed it on cleanup, which passes every code review. Run six back-to-back sessions, though, and the seventh silently gets no audio graph: transcription stops with no exception anywhere. The durable fix inverted the lifecycle: one AudioContext for the lifetime of the process, suspended between sessions instead of closed. Eight-plus consecutive sessions now stay healthy. The general lesson is that capture layers sit on platform resources with undocumented ceilings, and the ceiling only shows up under exactly the usage pattern, many sessions in one process, that a quick test never exercises. This failure returned no error at all, which is why [the observability post](/blog/silent-failures-observability) argues that the signal for this class has to be one you build on purpose. ## Audio you lose is gone The deepest difference from every other pipeline I run is that live audio cannot be re-requested. When the doc-intel [demo](/demo) drops a document mid-pipeline, a retry re-reads the file. When a transcription session drops mid-meeting, the meeting does not repeat itself. Whatever crossed the microphone during the gap either made it into a buffer or stopped existing. The first version buffered 80KB, about 5 seconds, of audio through a reconnect. Real networks blow through 5 seconds without trying. The current client buffers 2.2MB, around 45 seconds, and drains it at a paced rate after reconnecting, with chunk boundaries kept base64-safe so a resumed stream does not corrupt mid-frame. Sizing that buffer is a product decision disguised as a constant: it is the longest network outage the product promises to erase. The same no-retry property drove the telemetry. Every session tracks its reconnect count, whether churn triggered an automatic provider swap, the longest transcript gap in milliseconds, and whether both providers failed in one session. Three reconnects inside two minutes marks an upstream degraded, and the router moves traffic to the other provider. And because some corporate networks kill WebSockets outright, the transport itself falls back to HTTP with server-sent events when the socket cannot be established. None of this machinery adds a feature. All of it decides whether an hour-long transcript has holes. ## What I would do differently Everything above is shipped and running. This section is design judgment, stated as opinion. Compress on the wire. Raw 16kHz PCM is 256 kbps, and base64 framing pushes it to roughly 340. Opus at 24 kbps is transparent for speech and would cut the stream by an order of magnitude, which shrinks the reconnect buffer, and every bandwidth-driven decision downstream of it, proportionally. The silence-filtering that caused the idle-close saga was chasing savings Opus would have delivered without changing the traffic shape. Make pauses explicit in the protocol. The idle-close bug existed because silence was communicated by absence: the client stopped sending, and the provider read that as death. A session protocol where the client sends a cheap "paused" event, or a heartbeat designed in on day one, removes the entire class instead of patching around one provider's timeout. Treat the live transcript as the draft. A two-pass design writes a sidecar WAV during the session and re-transcribes it in batch afterwards, where a slower, more accurate pass can fix what the realtime pass garbled. Pavleur's codebase already has the sidecar recording built and the batch pass gated off; finishing that is the roadmap, because eventually-consistent transcripts beat pretending realtime output is final. Abstract the transport before the first provider integration, not after the third incident. The HTTP/SSE fallback was retrofitted; a transport interface from day one would have made it a configuration. ## Where this fits The provider lessons here rhyme with [surviving a model migration](/blog/surviving-a-model-migration): an STT vendor is a dependency with its own timeout policies and its own deprecation schedule, and the router-plus-fallback shape that survives model retirements is the same shape that survives a flaky realtime upstream. The receipts behind this post are unusual for this series in one way: they are mine end to end. The product is live at [pavleur.com](https://pavleur.com), the systems behind the rest of the series are in [the operator brief](/about), and the document pipeline this article kept contrasting against is answering questions at the [live demo](/demo), where a failed stage can simply retry, a luxury no microphone will ever grant.