Pricing checked on 2026-11-16.
Your voice agent doesn't fail because of noise. It fails because of a second voice.
Speech-to-text got very good at traffic, keyboards and air conditioning. It still has no idea which human it is supposed to listen to. When a colleague, a TV or the agent at the next desk talks near the mic, the STT transcribes that voice too, and your LLM answers it.
If you're:
- building a voice agent on Pipecat, LiveKit or your own stack,
- shipping a phone or call-center bot,
- or debugging an agent that interrupts itself on real calls,
this is the layer you're missing.
TL;DR
- Speech-to-text handles noise well. It still transcribes a second person talking near the mic, and that's what breaks voice agents.
- Only 2 of the 6 tools here are built to remove a second voice: Krisp Voice Isolation and Arctan Eigen. The open-source models remove noise or split speakers.
- On Krisp's open benchmark (265 recordings, 11 STT setups), WER fell from 23.24% to 6.24% (−73%) with Krisp Voice Isolation.
- In my own test (24 of those recordings, one small STT model), Krisp cut WER from 35.2% to 18.5% and the background-voice words in the transcript from 548 to 63.
- Krisp's API costs $0.18 per hour of audio and starts with $30 in free credits.
Short answer: my pick for production voice agents is Krisp Voice Isolation. It removes background voices and noise in real time, on CPU, with a self-serve API and an on-prem SDK, and it publishes an open benchmark (dataset, outputs and harness) that anyone can rerun. Only one other option in this list is built to remove a second voice (Arctan). The four open-source models remove noise or split speakers, which is a different job.
I ran the same 24 recordings through every tool I could run on my machine, transcribed each one with the same STT, and counted how many words from the other person ended up in the transcript.
Raw audio: 548 words from the background voice.
After Krisp: 63.
Why voice agents need voice isolation (and not just noise cancellation)
Noise is easy to remove because it doesn't look like speech. A second voice does.
That's the whole problem. A noise suppressor learns the shape of "not speech" and subtracts it. A colleague asking about lunch is speech, so it goes straight through to your STT.
Krisp puts a number on it. In its test for the Voice Isolation 2.5 release, 10 speech-to-text systems averaged 2.10% WER on clean single-speaker audio and 35.92% with a second person talking (release post).
On a voice agent, that turns into three bugs that look like logic bugs:
- Wrong transcripts. Words from the second voice land in the transcript, and the LLM answers things the caller never said.
- False interruptions. The agent hears a bystander, thinks the caller cut in, and stops talking.
- Broken turn-taking. Background speech looks like the caller is still talking, so the agent never takes its turn.
Your agent's logic works as designed.
It's reacting to a voice it should never have heard.
Noise suppression vs voice isolation vs speech separation
These three get mixed up all the time, and vendor landing pages don't help. For an agent, this is the difference that matters:
| Technique | What it removes | Output | Example in this list |
|---|---|---|---|
| Noise suppression | Non-speech sounds (fans, traffic, typing) | One stream, all voices kept | DeepFilterNet, DPDFNet, ClearerVoice (enhancement models) |
| Voice isolation | Noise and other people's voices | One stream: the main speaker only | Krisp Voice Isolation, Arctan Eigen |
| Speech separation | Nothing; it splits the mix | N streams, one per speaker. You pick | SepFormer WHAMR, ClearerVoice (separation model) |
Voice isolation sits first in the pipeline, before voice activity detection (VAD), turn detection and STT. If your agent uses a speech-to-speech model instead of an STT → LLM → TTS chain, it can sit in front of that model too (Krisp hasn't published results for that setup yet). Either way, every model after it hears one person.
How I tested
Same input, same STT, same scoring for every tool.
- Audio: 24 recordings from Krisp's open Voice Isolation Benchmark Dataset (CC BY-NC 4.0): 12 from home and office situations, 12 recorded inside working call centers. I picked only files with a labeled stretch where the second speaker talks alone. The dataset card says nothing is synthetically mixed.
- Input: the same 24 files for every tool, as 16 kHz mono 16-bit PCM. Two exceptions: Krisp's VIVA-pro model only accepted the original 32 kHz files, and SepFormer only works at 8 kHz, so it got a downsampled copy. Every output went to the STT at 16 kHz.
- STT: faster-whisper
base.enon CPU (greedy decoding, no VAD, so nothing hides the background voice). It's a small model, so absolute WER is higher than with a production STT. What matters is the gap between raw audio and each tool, because every tool goes through the same engine. - Metrics:
- WER against the dataset's hand-written transcripts.
- Leaked words: words transcribed during the windows where only the background voice is talking. The correct transcript there is empty, so every word is a word your agent should never have heard.
- CPU speed: processing time divided by audio length on my machine (8 cores, no GPU). Below 1.0 means faster than real time.
One disclosure: the dataset was recorded by Krisp. That's why I used their public data with my own STT and my own scoring script instead of their published numbers, and why I can share the raw transcripts and the scoring script on request.
Results: who actually removed the second voice
| Tool | WER (lower is better) | Background-voice words transcribed | vs raw | CPU speed (processing ÷ audio) |
|---|---|---|---|---|
| Raw audio (no processing) | 35.2% | 548 | — | — |
| Krisp Voice Isolation (VIVA-tel, API) | 18.5% | 63 | −89% | Cloud, real-time stream |
| Krisp Voice Isolation (VIVA-pro, API, 32 kHz) | 21.5% | 130 | −76% | Cloud, real-time stream |
| DeepFilterNet 3 | 29.5% | 282 | −49% | 0.14 |
| DPDFNet (dpdfnet4) | 32.6% | 498 | −9% | 0.46 |
| SepFormer WHAMR (louder stream kept) | 44.6% | 544 | −1% | 1.90 |
| ClearerVoice-Studio | Not scored (too slow on CPU) | — | — | 6.8+ (MossFormerGAN), 14.4 (FRCRN) |
| Arctan Eigen | Not tested | — | — | — |
ClearerVoice was too slow on CPU to run the full set (details in its section). 24 recordings, 3,198 reference words, 675 seconds where only the background voice talks. WER is high across the board because base.en is a small STT; read the gaps, not the absolute values.
Three things stood out.
Krisp cut word errors almost in half (35.2% → 18.5%) and removed about 9 of every 10 background words. On the 12 call-center recordings, the STT went from transcribing 234 words of the neighbouring agent to 4.
Noise suppressors don't solve the second-voice problem. DeepFilterNet removed about half of the background words, probably where the other voice was quiet enough to pass for noise, but it also doubled the words it deleted from the main speaker (88 → 184). DPDFNet left the second voice almost untouched, which is exactly what a noise suppressor should do.
Separation isn't isolation. SepFormer split the voices, but keeping the louder stream picked the wrong one often enough to make the transcript worse than doing nothing (17 of 24 files got worse).
Krisp wasn't perfect either. It made 5 of the 24 files slightly worse, by 0.5 to 2.4 WER points, all of them recordings where the raw audio had picked up almost none of the background voice. That matches what Krisp publishes about its own model: where there's no second voice, isolation costs a fraction of a point.
Comparison table
| Tool | Type | Removes a second voice? | How you run it | License / price | Sample rates | Latency |
|---|---|---|---|---|---|---|
| Krisp Voice Isolation | Voice isolation | Yes | Cloud API (WebSocket) or VIVA SDK on your servers | API $0.18/h, $0.12/h on Growth. SDK on Enterprise | API: VIVA-tel 8/16 kHz, VIVA-pro 32 kHz. SDK 8–32 kHz | 15 ms algorithmic (~20 ms added, SDK docs) |
| Arctan Eigen | Voice isolation | Yes (vendor claim) | SDK (Node, Python, Rust, C++), on-prem on paid plans | From $129/mo for 100,000 min | Not stated | <10 ms (vendor) |
| DeepFilterNet | Noise suppression | No | Open-source, Python/Rust/CLI | MIT or Apache-2.0, free | 48 kHz | Not stated in README |
| DPDFNet | Noise suppression | No | Open-source, ONNX on CPU | Apache-2.0, free | 8/16/48 kHz | ~20 ms first output, ~10 ms per block |
| ClearerVoice-Studio | Enhancement + separation toolkit | No (audio-only models) | Open-source, file-based | Apache-2.0, free | 16/48 kHz | Not stated |
| SepFormer WHAMR | 2-speaker separation | Splits, doesn't choose | Open-source (SpeechBrain) | Apache-2.0, free | 8 kHz only | Not stated |
1. Krisp Voice Isolation: my top pick for production voice agents
Krisp Voice Isolation keeps the main speaker and removes background voices and noise from a live audio stream. The current version, Voice Isolation 2.5, runs in real time on CPU and works in front of any STT engine. You can call it as a self-serve WebSocket API or run the VIVA SDK on your own servers, and Pipecat has a ready-made Krisp filter. If your agent also talks to callers in other languages, the same billing account covers Krisp's voice translation API.
# Full script in Krisp's API quickstart URL = "wss://streaming.krisp.ai/voice-isolation" async with websockets.connect(URL, additional_headers=auth) as ws: await ws.send(json.dumps({"model": "VIVA-tel", "sample_rate": 16000})) # then stream 30 ms PCM frames and read the cleaned frames back The API quickstart has the complete version, from API key to a cleaned WAV file.
My test: WER 35.2% → 18.5%, background-voice words 548 → 63 with the default API model (VIVA-tel). On Krisp's own open benchmark (265 recordings, 11 STT setups), WER fell from 23.24% to 6.24% (benchmark write-up). The same write-up also shows where it doesn't help: on phone calls with almost no second voice, WER went from 3.48% to 3.91%.
Pros
- Removes the second voice, not just noise.
- Public dataset and benchmark, so you can check the numbers yourself.
- Built for telephony codecs and 8 kHz audio, which is what phone agents receive.
Cons
- The on-prem SDK is Enterprise-only, with agreed pricing.
- Made for near-field audio (speaker within roughly 30 cm of the mic).
Pricing: Starter (pay-as-you-go): $0.18/h, with $30 in free credits valid for 12 months. Growth: $400/month with $400 of usage included, at $0.12/h. Same rate for VIVA-tel and VIVA-pro. SDK on Enterprise (pricing).
Best for: teams shipping voice agents on real calls (phone, call center, WebRTC) who need the agent to hear one person.
2. Arctan Eigen: a commercial option with an on-prem tier
Arctan Eigen is a real-time voice isolation product for voice agents. It goes after the same problem as Krisp. The homepage line is "Only the Primary speaker gets through." It ships first-party SDKs for Node, Python, Rust and C++, and a community-maintained Pipecat filter.
I didn't test it hands-on, so everything below comes from Arctan's own site.
Arctan claims 10 ms processing latency and a 57% WER reduction. Its homepage also shows a head-to-head WER chart against other isolation vendors, including Krisp. The methodology is "available on request" and the dataset isn't public, so I can't verify those numbers or compare them with the open benchmark above.
Pros
- Purpose-built for removing side talk, not just noise.
- On-premise deployment listed on the Startup and Pro plans.
- 14-day free trial.
Cons
- No public dataset or methodology behind the published numbers.
- Sample rates and the overage rate aren't stated on the site.
- The Pipecat filter is community-maintained and validates its license key over the network.
Pricing: Startup $129/month with 100,000 minutes, Pro $499/month with 500,000 minutes, Enterprise custom (pricing).
Best for: high, predictable monthly volume where a fixed plan with included minutes fits your budget.
3. DeepFilterNet: the best free baseline in my test
DeepFilterNet is a widely used open-source speech enhancement framework (about 5,000 GitHub stars): full-band 48 kHz noise suppression, Python and Rust libraries, a prebuilt deep-filter binary and a LADSPA plugin.
pip install deepfilternet from df import enhance, init_df model, df_state, _ = init_df() clean = enhance(model, df_state, noisy_audio) On my Windows machine the Python package failed to build (it compiles a Rust extension), so I used the official prebuilt CLI.
My test: WER 35.2% → 29.5%, background-voice words 548 → 282. It was also the fastest model I ran: 0.14 seconds of CPU per second of audio, resampling included. The catch is the other column: it deleted twice as many of the main speaker's words as raw audio did (88 → 184). It doesn't know which voice matters, so it trims whatever sounds like interference.
Pros
- Permissive license (MIT or Apache-2.0).
- The fastest option on CPU in my test.
- A prebuilt
deep-filterCLI on the releases page, so you can skip the Python build.
Cons
- Built for noise. It removed about half the background words, but took some of the main speaker with them.
- 48 kHz only, so 8 kHz phone audio has to be resampled.
- No release since v0.5.6 (August 2023).
Pricing: free.
Best for: a free baseline, and self-hosted agents where the problem is noise rather than people.
4. DPDFNet (Ceva): open-source noise suppression with a streaming API
DPDFNet is a family of causal noise suppression models from Ceva, built on DeepFilterNet2 with dual-path RNN blocks. It's the most recently released open-source option here (PyPI 0.6.0, July 2026) and ships a documented streaming API.
import soundfile as sf import dpdfnet audio, sr = sf.read("noisy.wav") clean = dpdfnet.enhance(audio, sample_rate=sr, model="dpdfnet4") sf.write("clean.wav", clean, sr) For live audio, dpdfnet.StreamEnhancer keeps the RNN state between chunks. The first output arrives after about 20 ms, then about 10 ms per block.
My test: WER 35.2% → 32.6%, background-voice words 548 → 498. It kept the main speaker intact (fewer deleted words than raw audio) and left the second voice almost untouched. That's correct behaviour for a noise suppressor, and the reason it can't fix a second-voice problem. It ran at 0.46 seconds of CPU per second of audio with the dpdfnet4 model.
Pros
- Apache-2.0, runs on CPU through ONNX, models are 8–14 MB.
- Native 8, 16 and 48 kHz, so telephony audio works without resampling.
- Streaming API with documented latency.
Cons
- Removes noise, not voices. A second speaker goes straight through.
- No documented integration with voice agent frameworks; you write the glue.
- Small community (under 200 GitHub stars).
Pricing: free (Apache-2.0). You pay for the CPU.
Best for: self-hosted agents in noisy places where a second voice is rare, like a caller on the street.
5. ClearerVoice-Studio: a research toolkit, not a real-time filter
ClearerVoice-Studio from Alibaba's speech lab bundles speech enhancement, speech separation, super-resolution and target-speaker extraction behind one ClearVoice interface. Its best 16 kHz enhancement model, MossFormerGAN_SE_16K, reports a PESQ of 3.47 on VoiceBank+DEMAND.
from clearvoice import ClearVoice cv = ClearVoice(task="speech_enhancement", model_names=["MossFormerGAN_SE_16K"]) out = cv(input_path="noisy.wav", online_write=False) cv.write(out, output_path="clean.wav") The catch for agents: the target-speaker model that would pick one voice is audio-visual. It needs video of the speaker, which a phone call doesn't have. The API is file-based, and real-time use isn't documented.
My test: I couldn't score it. On my CPU, out of the box, MossFormerGAN_SE_16K didn't finish a 132-second recording in 15 minutes, and the lighter FRCRN_SE_16K took 433 seconds to process a 30-second clip. That's 7 to 14 times slower than real time, so the 24-file test would have taken about a day. On CPU, at least, it's a tool for offline batch work, not for sitting in front of a live call.
Pros
- Many models under one Apache-2.0 toolkit.
- Benchmark tables in the README (PESQ for enhancement, SI-SNRi for separation).
- Weights download automatically from Hugging Face.
Cons
- No shipped audio-only "keep the main speaker" model.
- File-based, with no documented streaming API.
- In my install, its pinned dependencies (NumPy below 2, an older soundfile) clashed with the other packages.
Pricing: free.
Best for: offline cleanup of recordings and research, not live agents.
6. SepFormer WHAMR (SpeechBrain): separation without a decision
SepFormer WHAMR is a transformer that splits a noisy, reverberant two-speaker mix into two streams. It scores 13.7 dB SI-SNRi on the WHAMR! test set.
from speechbrain.inference.separation import SepformerSeparation model = SepformerSeparation.from_hparams(source="speechbrain/sepformer-whamr", savedir="pretrained_models/sepformer-whamr") sources = model.separate_file(path="mix_8k.wav") # [batch, time, 2] That last comment is the problem for an agent. You get two streams and no label saying which one is the caller. In my test I kept the louder stream, since the person closest to the mic is usually louder, but that's a heuristic you'd have to own in production.
My test: WER 35.2% → 44.6%, background-voice words 548 → 544. Keeping the louder stream chose the wrong speaker often enough that 17 of 24 files came out worse than raw audio. Of the models I scored, it was the only one slower than real time on my CPU (1.9 seconds of processing per second of audio), so on CPU it can't keep up with a live call.
Pros
- Actually separates voices, not just noise.
- Apache-2.0, a few lines with SpeechBrain.
Cons
- 8 kHz only, trained on read speech (WSJ0).
- Doesn't choose the primary speaker.
- No documented streaming API; it was released in 2021.
Pricing: free.
Best for: offline speaker separation and research baselines.
How to choose a voice isolation API for your agent
Start from the question your agent fails on.
- "It answers things the caller never said." You have a second-voice problem. Use voice isolation: Krisp's API to start, the VIVA SDK if audio must stay on your servers.
- "It only struggles in noisy places." A noise suppressor may be enough. DPDFNet streams on CPU at 8/16/48 kHz.
- "I can't send audio to a third-party cloud." Krisp's VIVA SDK or Arctan's on-prem tier, or an open-source suppressor if you can live without voice removal.
- "I'm cleaning recordings, not live calls." Offline tools like ClearerVoice or SepFormer are fine, and latency doesn't matter.
Whatever you pick, measure it on your audio. Krisp's own benchmark shows isolation can cost a fraction of a point on phone calls with no second voice, so test before you turn it on for all traffic.
FAQ
What is a voice isolation API?
An API that takes live audio and returns only the main speaker's voice, with background noise and other people's voices removed. It sits before VAD and speech-to-text in a voice agent pipeline, or in front of a speech-to-speech model.
What's the difference between voice isolation and noise cancellation?
Noise cancellation removes non-speech sounds. Voice isolation also removes other voices. For a voice agent, the second voice is what causes wrong transcripts and false interruptions.
Do I still need voice isolation if my STT handles noise well?
Yes, if a second person can be near the mic. In Krisp's test, 10 STT systems averaged 2.10% WER on clean audio and 35.92% with a second voice. STT engines are robust to noise, not to competing speech.
Is there a free, open-source voice isolation model?
Among the tools here, the open-source ones are noise suppressors (DeepFilterNet, DPDFNet, ClearerVoice) or speaker separators that don't pick a speaker (SepFormer). Krisp's API has $30 in free credits, which covers about 166 hours of audio at $0.18/h.
Does voice isolation add latency?
A little. Krisp states 15 ms of algorithmic latency for VI 2.5 (about 20 ms added, per the SDK docs). DPDFNet's streaming mode returns its first output after about 20 ms.
Does voice isolation work with LiveKit and Pipecat?
Krisp VIVA is available in both: Pipecat has a ready-made Krisp VIVA filter, and LiveKit lists Krisp VIVA among its noise cancellation options.
The shift
Noise used to be the problem. Modern STT mostly handles it.
The problem now is the second voice, and it needs its own layer in front of your agent.
Looking for technical content for your company? I can help — LinkedIn · kevinmenesesgonzalez@gmail.com



