Open-Source Speech-to-Text in 2026: Lessons from Transcribing 2,750 Calls a Day

Open-source speech-to-text had a remarkable eighteen months. The Hugging Face Open ASR Leaderboard, which Whisper once owned by default, now has five models within a single WER point of each other at the top — from Cohere, IBM, Alibaba and labs most teams had never evaluated before 2026. Word error rates that cost $1 an hour from an API in 2024 are now available in Apache-licensed checkpoints that fit on a workstation GPU.

We just finished moving a consumer-lending collections operation — roughly 2,750 recorded calls a day, about 90 hours of telephony audio — from a hosted multimodal LLM onto a fully self-hosted open-source pipeline. The monthly transcription-and-analysis bill went from about $1,680 to roughly $55 of electricity. But the more useful outcome was what the evaluation taught us, because nearly every headline fact about open STT turned out to be misleading for this workload. If you are picking a model in 2026, the process matters more than the pick.

The best open-source speech-to-text models in 2026

The short answer: there is no single best open-source speech-to-text model in 2026. Whisper large-v3 has the best tooling, the leaderboard leaders win on clean English, and a purpose-trained regional model can beat all of them on your actual audio. The table below is the map; the rest of this post is how to choose.

These are the models that came up in our evaluation or dominate the current leaderboard, with the facts that end up mattering in production:

Notice what the table implies: the "best" column does not exist. The leaderboard leaders are benchmarked overwhelmingly on clean, English, wideband audio. Our workload was none of those things — 8kHz telephony, Hindi and Hinglish code-switching with heavy English loanwords, Marathi from one region's borrowers, two speakers talking over each other about money.

Whisper and its alternatives on real telephony audio: what won

We built a seven-fact benchmark from a real call: a settlement amount, a duration ("4 months"), a vehicle model, the distinction between a CIBIL score discussion and a settlement, an address, a mobile number, and — critically — whether the model hallucinated content in languages nobody spoke. Then we ran the candidates over the same channel-split, loudness-normalized audio.

Canary-Qwen never made it to the bench: NVIDIA's own model card says English-only, a fact you will not learn from a leaderboard screenshot. That is the first trap — public ASR leaderboards are implicitly English leaderboards, and "state of the art" claims simply do not transfer to code-switched telephony speech.

Whisper large-v3, served through vLLM, looked great on paper: native timestamps, mature tooling, and we measured 29.5x request concurrency on a single RTX 4080 using under 3GB of weights. On our audio it fell apart in two specific ways. Vocabulary biasing through the prompt field — a technique that measurably helped Voxtral — destabilized Whisper's decoding badly enough to lose the settlement amount; dropping the bias recovered 38 segments. And on a long silence it entered the classic repetition loop, emitting the same segment ten times for 27 seconds. You can transliterate Devanagari deterministically, but you cannot recover a dropped rupee amount.

Voxtral Small 24B was the most accurate model we tested — zero hallucination, every key fact right. It also needs about 55GB of memory in fp16, no official quantized checkpoint existed at the time of our evaluation, and it therefore does not fit a 16GB card. Its little sibling, Voxtral Mini 3B, fits — and turned "4 months" into "4 minutes", dropped the vehicle, and hallucinated English and Spanish on the quieter channel.

The winner was AI4Bharat's IndicConformer 600M multilingual — MIT-licensed, purpose-trained on 22 Indian languages including exactly the code-switched banking vocabulary our calls are full of. It produced 148 clean segments where Whisper managed 50, recovered six of the seven facts, and drifted into foreign scripts exactly zero percent of the time. It is on no global leaderboard. It was better than everything that is.

One configuration note that generalizes: IndicConformer supports both CTC and RNNT decoding, and the RNNT decoder was dramatically better on our audio — the CTC path produced garbage on some segments. Decoder choice is a real variable, not a footnote.

The architecture: two stages, and diarization for free

The old pipeline sent each call's audio to a hosted multimodal LLM three separate times — summary, QA scoring, disposition — and asked for structured JSON straight from audio. That design had a subtle failure mode that ultimately mattered more than cost: audio LLMs are confidently unreliable about time. They fabricated plausible-looking timestamps beyond the end of the call and emitted templated "conversational arc" entries regardless of content. Accurate about what, inventive about when.

So the new pipeline separates hearing from thinking:

01Stage A (CPU): IndicConformer 600M, RNNT decoder
02  ffmpeg channel split  →  officer / borrower tracks
03  adaptive-threshold VAD per channel  →  segment timestamps
04  → call_transcripts collection (MongoDB)
05
06Stage B (GPU): Gemma-4-12B QAT INT4 via vLLM
07  reads stored transcripts, never audio
08  → summary, 9-criteria QA score, disposition JSON

Three decisions in that diagram carry most of the value.

Channel splitting is free, perfect diarization. Telephony recordings are often true dual-channel — agent on the left, customer on the right. One ffmpeg invocation gives you speaker attribution no diarization model can match. Verify it is real stereo, not dual-mono, by comparing L−R against L+R loudness; file metadata lies, and every diarization model we tried inverted speaker labels somewhere.

The ASR stage runs on CPU — deliberately. IndicConformer transcribes a minute of audio in 7.1 seconds of CPU time; the day's whole volume needs about 11 CPU-hours against a 24-hour budget. Moving it off the GPU freed all 16GB for the small-language-model reasoning stage, whose vLLM continuous batching delivers 6 seconds per call at concurrency 8 — a 5x headroom over daily volume. The same model on a single-stream server ran 111 seconds per call, 3.5x too slow. Batching architecture, not model choice, decided whether the hardware was sufficient.

Transcripts became a first-class store. Once transcripts persist in their own collection, re-running analysis with a better prompt costs text-only tokens, recordings can expire upstream without loss, and — one production bug we fixed on the way — raw borrower conversations stop leaking into any list API that lacks field projection.

Self-hosted vs hosted speech-to-text: the cost math, honestly

Two honest caveats. First, the restructuring mattered more than the hosting decision: simply transcribing once and analyzing as text — still fully hosted — would have halved the bill before any hardware was bought. When we priced a straight provider swap under the old one-shot architecture, the "cheaper" provider came out 25 percent more expensive. Second, $55 is marginal cost. It excludes the engineering weeks to build and harden the pipeline, and the ongoing ownership of it. Self-hosting wins at sustained volume, with in-house engineering, or when the audio is too sensitive to leave the building — collections calls are all three. At a few hundred calls a day, use an API and move on.

The gotchas that cost us days

• A --device cpu flag does not mean CPU inference. ONNX Runtime selects its execution provider independently of torch and silently grabbed the GPU, making an early benchmark meaningless. Set CUDA_VISIBLE_DEVICES="" when you need a true CPU number.
• Never install onnxruntime and onnxruntime-gpu together — they shadow the same import path and the CPU build wins silently. No error, no GPU, wrong conclusions.
• Match ONNX Runtime's CUDA major version to your torch build. The mismatch produces "library not found" errors that point everywhere except the actual cause.
• Python 3.14 was too new: no CUDA wheels, so torch silently installed CPU-only. Python 3.12 via pyenv fixed a problem that looked like a hardware issue.
• Prompt biasing is not portable across model families. The identical vocabulary-bias approach helped Voxtral and broke Whisper. A/B it per model; never assume it transfers.
• Check finish_reason on every LLM call. At the output-token limit, a model can burn its entire budget and return an empty string that looks exactly like success. The legacy pipeline had recorded silent empty analyses for months.
• Keep models resident. A 15-minute cron reloading a 2.4GB model wastes 7 percent of runtime purely on loading; a resident worker loop with systemd Restart=always is the fix.

How to run this decision for your own workload

The generalizable method, in five steps: build a small benchmark of facts that must survive transcription from your real audio, not a WER script on clean speech. Check language and domain coverage before rank — a model trained on your language's code-switching patterns will beat a leaderboard champion. Test decoder variants and prompt biasing per model. Decide batch versus streaming honestly; batch pipelines are dramatically simpler and most analytics workloads do not need live transcripts. And price the architecture before the provider — transcribe-once-analyze-many changes the economics more than any vendor switch.

FAQ

Is Whisper still the default choice in 2026?

It is the default starting point, not the default answer. Its ecosystem — faster-whisper, whisper.cpp, WhisperX — remains unmatched, and for clean English or broad multilingual coverage it is excellent. But it no longer tops accuracy benchmarks, and for specific language families a purpose-trained regional model can beat it outright. Evaluate it; do not assume it.

What about diarization and word timestamps?

If your audio is telephony, check for dual-channel first — channel splitting beats every diarization model and costs one ffmpeg call. Otherwise pyannote 3.x is the standard bolt-on, WhisperX packages alignment and diarization neatly, and 2026's trend is diarization moving into the model itself. Models like IndicConformer emit no timestamps at all — budget for a VAD-based segmentation layer if you need them.

What hardware does a self-hosted pipeline actually need?

Less than you think, if you separate stages. Our entire 2,750-calls-a-day operation runs on two workstations with 16GB consumer GPUs — ASR on CPU cores, LLM analysis on the GPU. The binding constraint on the GPU was KV-cache capacity for batching, not model weights. Rented equivalents run under $1/hour on AWS; the leaderboard-tier models are 2–3B parameters and undemanding.

When should we stay on a hosted API?

Below roughly a thousand calls or podcast-hours a month, almost always. Hosted APIs now cost $0.15–0.45 per hour of audio, and the hidden costs of self-hosting — building batching, monitoring, VAD, redeployment — are measured in engineer-weeks. The crossover comes with sustained volume, data-residency requirements, or when your audio would need the expensive multimodal tiers to handle a language the cheap tiers do not.

This evaluation-and-migration is a service we run end to end — model benchmarking on your real audio, pipeline build, and handover — as part of our voice AI practice, alongside the LLMOps work that keeps it observable in production. If your transcription bill has crossed four figures a month, it is very likely paying for an architecture, not a model — talk to us.