Speech-to-text was the biggest line in our model bill: about 55% of 14 days' model spend, ahead of video understanding (37%) and the text model that writes the summary (9%). So that is what we changed first.
This is from a link-reading tool I build, LinkDigest, which turns Chinese short-video links into transcripts. The comparison below is one clip, so treat the numbers as a starting point for your own test, not a benchmark.
Five models, one 90-second Mandarin clip
A fast, multi-speaker comedy sketch with a few domain words, sent to every model through the same API gateway. Cost is what the gateway billed.
| Model | How it is called | Cost | Time | Result |
|---|---|---|---|---|
| gpt-4o-mini-transcribe (what we used) | transcription endpoint | $0.0021 | 5.5 s | baseline |
| qwen3.8-omni-flash | chat + audio input | $0.00058 | 14.6 s | as clean as the baseline, domain words right |
| whisper-large-v3 | transcription endpoint | $0.00068 | 5.0 s | several domain words wrong |
| gemini-3.5-flash-lite | chat + audio input | $0.0011 | 4.3 s | more wrong characters |
| gemini-2.5-flash-lite | chat + audio input | $0.0015 | 7.2 s | unusable: about 220 characters were spoken, it wrote 1,510, the second half a loop |
The last row is the point. A chat model is not a transcription endpoint. The endpoint returns what it heard. A chat model may complete, apologise, or repeat one sentence for a page, and every one of those comes back as an ordinary-looking answer with HTTP 200.
Calling a chat model with audio
An OpenAI-compatible request, the audio base64-encoded inside the message:
{ "model": "qwen/qwen3.8-omni-flash", "temperature": 0, "max_tokens": 4000, "messages": [{ "role": "user", "content": [ {"type": "text", "text": "Transcribe the speech word for word, in the language it is spoken in. Do not summarise or translate. If nobody speaks, output exactly <no speech>"}, {"type": "input_audio", "input_audio": {"data": "<base64 mp3>", "format": "mp3"}} ] }] } Two things in the prompt matter: "in the language it is spoken in" (otherwise you may get a translation), and a fixed marker for silence. Without the marker, a silent chunk comes back as a paragraph.
Three checks before a transcript is kept
Audio is cut into 170-second chunks; each chunk's answer goes through three checks:
- Is it an apology or a refusal? A short answer starting with "I'm sorry", 抱歉 and the like is not a transcript.
- Is the pace possible? More than 12 spoken units a second (one per Chinese character, one per English word) is not speech. Fast Mandarin is about 6 characters a second.
- Is it looping? The same 12 characters 8 or more times is a repeat, at any length.
import re REFUSAL = re.compile(r"^\s*(抱歉|对不起|很抱歉|I'?m sorry|Sorry|I can(?:'|no)t|As an AI)", re.I) MAX_UNITS_PER_SECOND = 12.0 # fast speech: ~6 Chinese characters or ~4 English words a second def units(text: str) -> int: """Spoken units: one per Chinese character, one per run of letters or digits.""" cjk = len(re.findall(r"[㐀-鿿]", text)) words = len(re.findall(r"[A-Za-z0-9]+", text)) return cjk + words def looks_like_transcript(text: str, seconds: float) -> tuple[bool, str]: if REFUSAL.match(text) and len(text) < 120: return False, "an apology or refusal, not a transcript" if seconds > 0 and units(text) / seconds > MAX_UNITS_PER_SECOND: return False, f"{units(text) / seconds:.1f} units a second, faster than anyone speaks" compact = re.sub(r"\s+", "", text) seen: dict[str, int] = {} for i in range(len(compact) - 11): gram = compact[i:i + 12] seen[gram] = seen.get(gram, 0) + 1 if seen[gram] >= 8: return False, "the same 12 characters 8 times or more: a loop" return True, "usable" SAID = "那些小时候幻想的职业,我以后一定要当一个特别文艺的文学编辑。" cases = [ ("normal transcript, 30 s", SAID, 30), ("the model apologises", "I'm sorry, but I can't process this audio.", 30), ("too fast: 660 chars in 30 s", "今天我们来聊聊职业规划" * 60, 30), ("a loop", "对的对的对的对的。" * 30, 90), ("fast English", "so today we are going to look at three ways to cook tofu " * 6, 30), ] for name, text, seconds in cases: ok, why = looks_like_transcript(text, seconds) print(f"{name}: {'keep' if ok else 'fall back'} ({why})") Output:
normal transcript, 30 s: keep (usable) the model apologises: fall back (an apology or refusal, not a transcript) too fast: 660 chars in 30 s: fall back (22.0 units a second, faster than anyone speaks) a loop: fall back (the same 12 characters 8 times or more: a loop) fast English: keep (usable) Three more cases also fall back without a check: an HTTP error or a timeout (we give the chat model 90 seconds), finish_reason == "length" (cut off), and the silence marker. Deciding that a chunk is silent stays with the transcription endpoint; the chat model never gets to drop one.
The fallback: worst case is the old behaviour
Any chunk that fails a check is transcribed again by the old endpoint. So the worst case is what we had before, a little slower and with one extra call. Both calls are written to the cost ledger, and so is the model whose words were kept, which is how you see the fallback rate.
The first day in production
- Over the first 30-odd real reads (Douyin, Bilibili and Xiaohongshu videos, the longest about five minutes in two chunks), one transcript came from the old endpoint and the rest from the chat model.
- The chat model is slower per chunk (13–15 s for the 90-second clip against 5.5 s). Chunks run in parallel, so a five-minute video's speech took 30–37 s.
- A whole five-minute read, transcript plus summary, cost about $0.003 in model fees.
What I'd keep
- Look at where the money goes before picking a model to replace.
- Compare on one clip through one gateway: cost, time and the obvious errors, written down.
- Give a chat model doing transcription code checks for apologies, impossible pace, loops, cut-offs and errors, and fall back on every one.
- Leave the silence decision to the transcription endpoint.
- Log which model's words were kept, and judge the change on real traffic, not on the test clip.