If you use faster-whisper or openai-whisper with an initial_prompt, try this sometime. Record two seconds of a fan, or a keyboard, or nothing in particular, loud enough to pass whatever loudness check you have. Transcribe it.

There is a decent chance you get your prompt back.

In YazSes, the offline dictation tool I work on, the built-in prompt is one short sentence:

BUILTIN_PROMPT = "The app is called YazSes."

It is there so Whisper spells the app's name right. And on a noisy burst with no speech in it, Whisper would sometimes transcribe it as... "The app is called YazSes." Typed straight into whatever window you had open. (#598 has the details.)

Why this happens

Whisper treats initial_prompt as if it were the text that came just before the audio. My working explanation (a hypothesis, not something I have proven inside the model) is that when the audio carries no words, the easiest thing for the decoder to produce is the text it was just shown. Loud noise is the worst case: a quiet clip gets dropped by the energy gate before Whisper sees it, but a loud one gets through.

Why the obvious fix doesn't work

Whisper reports a no_speech_prob for each segment. Surely you just drop segments where it is high?

I measured it, with the small, base.en and tiny.en models on synthetic noise bursts (white, brown and low-frequency noise and clicks, half a second to three seconds). On the echoes, no_speech_prob ranged from 0.43 to 0.97. On a public-domain speech sample it was 0.02 to 0.27.

Two problems with cutting on that:

  • faster-whisper's default no_speech_threshold is 0.6. Every echo between 0.43 and 0.6 sails straight through the default settings.
  • The gap between 0.27 and 0.43 comes from one speech sample and some synthetic noise. I would not bet a user's words on a cutoff placed inside a gap that narrow, measured that thinly.

And it wasn't only the built-in prompt. A personal vocabulary list came back too: "Ydotool, ydotool, Mohsen, Wayland." Whatever you prime the model with is what it reaches for when there is nothing to hear.

The fix: ask a different model

The fix is to stop asking Whisper whether there was speech, and ask an independent detector before decoding.

faster-whisper already bundles Silero VAD, so it costs no new dependency. Before a burst is decoded, YazSes asks Silero whether it holds any speech. If Silero positively says no, the burst is discarded and logged as discard_reason = "no_speech".

Energy gate, then Silero VAD; only a positive 'no speech' discards the burst

The most important design decision is the direction it fails in:

def holds_no_speech( audio: Any, sample_rate: int, *, detector: Detector | None = None ) -> bool: """True only when the detector *positively* found no speech. Never raises. ``None`` from the detector (unavailable, wrong sample rate) and any exception both read as "cannot tell", which is ``False``: the burst is decoded. """ try: verdict = (detector or _default_detector)(audio, sample_rate) except Exception: # noqa: BLE001 - a gate must never break dictation  return False return verdict is True

It fails open. If Silero can't run, can't load, or gets audio at the wrong rate, the burst is decoded exactly as before. Think about the costs: a phantom sentence is annoying and you delete it. Silently swallowing words the user actually said is much worse, because they may not notice until later. A gate that guards against the small error must never cause the big one.

Choosing the threshold

Silero's default speech threshold is 0.5. I used 0.3, and here is the evidence, including its limits:

  • I cut 22 single-word bursts from the public-domain JFK speech sample and made 27 seeded noise bursts.
  • At 0.3: 0 words dropped, 0 noise bursts passed.
  • At 0.5: 2 quiet words were lost.
  • With a crude "whisper" proxy (random-phase audio), 3 of 22 words were lost at 0.3.

That is synthetic audio and a small set. I have not measured it on real whispered, accented or dysarthric speech, and those are exactly the users a dictation tool should care about most. So the check is skipped in the modes built for them: whispered/low-effort mode, the dysfluency-friendly mode and the quiet sotto-voce channel. It can also be turned off:

[accessibility] speech_presence_check = true # the default since v2.43.2

What I took away

  • Loudness can't tell a person from a fan. An energy gate is a first filter, not a speech detector.
  • Don't ask the model that is failing whether it is failing. Its own no_speech_prob let echoes through at default settings. A second, independent model that never saw the prompt answers the question more cleanly.
  • Decide which way a gate should fail before you write it. Then write the test for the failure, not just the happy path.

If you run Whisper with a prompt in any app (meeting notes, voice notes, subtitles), check what it returns on two seconds of loud noise. It takes a minute and you may be surprised.

The fix shipped in YazSes v2.43.2 (release notes). YazSes is offline, open source and Apache-2.0: github.com/MSKazemi/yazses.

I wrote this with help from an AI assistant for drafting and editing; the bug, the measurements and the design choices are mine, and I checked every fact against the code and release notes.