Pavleur, the live work assistant I build and run, streams meeting audio into a transcript in real time and feeds that transcript to a model. Getting a speech-to-text provider to return words took a few days. Keeping a transcript alive through a real hour-long meeting, on real networks, across two providers with different failure moods, is where the engineering went. This post is that pipeline, with first-party receipts: it is my own product, live at pavleur.com, so nothing here hides behind an NDA.
The honest summary up front: transcription is the easy third of a transcription product. The rest is failure handling, and real audio finds failures a document pipeline never meets, because live audio is the one input you cannot ask for again.
The clean pipeline, as designed
On the whiteboard the system is tidy. An Electron client captures the microphone and, for meetings, the system audio too. Audio is resampled to 16kHz, mixed to mono, and converted to 16-bit PCM, then shipped in 2,048-sample chunks, about 128 milliseconds each, over a WebSocket to a proxy on Fly.io.
The proxy exists because API keys, quotas, and plan enforcement do not belong in a desktop binary. It also owns the routing decision: English sessions go to Deepgram Nova-3, which gets 50 concurrent slots; other languages go to ElevenLabs Scribe v2, which gets 6. Both stream interim results back, so words appear while the speaker is mid-sentence. Commit boundaries come from voice activity detection: a segment finalizes after one second of silence, with 200 milliseconds minimum speech and 500 milliseconds minimum silence so breaths and clicks do not become utterances.
That design is real and it ships. It is also maybe a third of the code that actually runs. The rest accumulated one production surprise at a time.
The provider that hangs up on silence
To save bandwidth, the client filters silence before it ever reaches the wire. Meetings are full of pauses, so most of the time nothing is being sent, which is exactly the optimization you would make.
ElevenLabs' realtime endpoint idle-closes the upstream connection after roughly 16 seconds without an audio chunk. Put those two sensible decisions together and the product does something absurd: every time the room goes quiet for a stretch, the provider hangs up, the proxy reconnects, and the user watches a "reconnecting" indicator flicker in the middle of a perfectly healthy session.
The fix is a keepalive that sends frames of silent PCM every 8 seconds during pauses. Zeroed samples never trip VAD, so they cannot corrupt the transcript, but they reset the provider's idle clock. It works, and it is load-bearing, and it is the kind of fix nobody diagrams: my bandwidth optimization was the provider's liveness signal, and neither side was wrong.
The browser resource that ran out
The capture layer died in a way that took real effort to see. Chromium hard-caps AudioContexts at roughly 6 per page, and closing one does not free the slot immediately, because the release rides on garbage collection. Pavleur created a fresh AudioContext per session and closed it on cleanup, which passes every code review. Run six back-to-back sessions, though, and the seventh silently gets no audio graph: transcription stops with no exception anywhere.
The durable fix inverted the lifecycle: one AudioContext for the lifetime of the process, suspended between sessions instead of closed. Eight-plus consecutive sessions now stay healthy. The general lesson is that capture layers sit on platform resources with undocumented ceilings, and the ceiling only shows up under exactly the usage pattern, many sessions in one process, that a quick test never exercises. This failure returned no error at all, which is why the observability post argues that the signal for this class has to be one you build on purpose.
Audio you lose is gone
The deepest difference from every other pipeline I run is that live audio cannot be re-requested. When the doc-intel demo drops a document mid-pipeline, a retry re-reads the file. When a transcription session drops mid-meeting, the meeting does not repeat itself. Whatever crossed the microphone during the gap either made it into a buffer or stopped existing.
The first version buffered 80KB, about 5 seconds, of audio through a reconnect. Real networks blow through 5 seconds without trying. The current client buffers 2.2MB, around 45 seconds, and drains it at a paced rate after reconnecting, with chunk boundaries kept base64-safe so a resumed stream does not corrupt mid-frame. Sizing that buffer is a product decision disguised as a constant: it is the longest network outage the product promises to erase.
The same no-retry property drove the telemetry. Every session tracks its reconnect count, whether churn triggered an automatic provider swap, the longest transcript gap in milliseconds, and whether both providers failed in one session. Three reconnects inside two minutes marks an upstream degraded, and the router moves traffic to the other provider. And because some corporate networks kill WebSockets outright, the transport itself falls back to HTTP with server-sent events when the socket cannot be established. None of this machinery adds a feature. All of it decides whether an hour-long transcript has holes.
What I would do differently
Everything above is shipped and running. This section is design judgment, stated as opinion.
Compress on the wire. Raw 16kHz PCM is 256 kbps, and base64 framing pushes it to roughly 340. Opus at 24 kbps is transparent for speech and would cut the stream by an order of magnitude, which shrinks the reconnect buffer, and every bandwidth-driven decision downstream of it, proportionally. The silence-filtering that caused the idle-close saga was chasing savings Opus would have delivered without changing the traffic shape.
Make pauses explicit in the protocol. The idle-close bug existed because silence was communicated by absence: the client stopped sending, and the provider read that as death. A session protocol where the client sends a cheap "paused" event, or a heartbeat designed in on day one, removes the entire class instead of patching around one provider's timeout.
Treat the live transcript as the draft. A two-pass design writes a sidecar WAV during the session and re-transcribes it in batch afterwards, where a slower, more accurate pass can fix what the realtime pass garbled. Pavleur's codebase already has the sidecar recording built and the batch pass gated off; finishing that is the roadmap, because eventually-consistent transcripts beat pretending realtime output is final.
Abstract the transport before the first provider integration, not after the third incident. The HTTP/SSE fallback was retrofitted; a transport interface from day one would have made it a configuration.
Where this fits
The provider lessons here rhyme with surviving a model migration: an STT vendor is a dependency with its own timeout policies and its own deprecation schedule, and the router-plus-fallback shape that survives model retirements is the same shape that survives a flaky realtime upstream. The receipts behind this post are unusual for this series in one way: they are mine end to end. The product is live at pavleur.com, the systems behind the rest of the series are in the operator brief, and the document pipeline this article kept contrasting against is answering questions at the live demo, where a failed stage can simply retry, a luxury no microphone will ever grant.