A single scalar metric tells you a model ranked well on a benchmark leaderboard. It does not tell you why it failed in production, how confident the measurement is, or where the acoustic distortion actually occurred.
In music source separation, deep learning models are celebrated for high Signal-to-Distortion Ratio (SDR) while silently leaking loud drum snare hits into isolated vocal tracks. In conversational voice AI, speech-to-text models achieve low Word Error Rate (WER) while remaining completely blind to temporal conversational dynamics: speaker overlap, interruption handling, and turn-taking latency.
This article introduces an open-source, diagnostic evaluation methodology developed at ComposerMind Labs. By combining deterministic mixture synthesis (mixforge-audio), directional inter-stem bleed forensics (stemcheck), and millisecond-level conversational turn-taking analytics (turnscope), this framework moves audio evaluation beyond aggregate scalar scores into actionable, reproducible engineering diagnostics.
+-------------------------------------------------------------------------------------------------+
| COMPOSERMIND AUDIOEVALUATION WORKFLOW |
| |
| [ PILLAR 1: CONTROLLED GROUND-TRUTH MIXTURES ] |
| * mixforge-audio (v0.1.0) --> Calibrated Inter-Stem Bleed (-10 to -40 dB), Deterministic Sum |
| |
| [ PILLAR 2: DIAGNOSTIC ACOUSTIC FORENSICS ] |
| * stemcheck (v0.1.0) --> Least-Squares Bleed Matrix, Empirical Chance-Floor Calibration, |
| Linear Mixture Reconstruction Completeness (dB) |
| |
| [ PILLAR 3: CONVERSATIONAL DYNAMICS ANALYTICS ] |
| * turnscope (v0.1.0) --> Millisecond Handover Latency, Speaker Overlap Ratio, |
| Agent Interruptions, Barge-Ins, and Dead-Air Profiling |
+-------------------------------------------------------------------------------------------------+
—————————————————————————————————————————————
- The Fallacy of Aggregate Leaderboard Metrics Modern machine learning research relies heavily on scalar leaderboard metrics to track incremental progress. In audio and speech processing, two metrics dominate: • Signal-to-Distortion Ratio (SDR) for audio source separation (BSS Eval). • Word Error Rate (WER) for Automatic Speech Recognition (ASR). While these numbers provide convenient rankings, they systematically obscure localized, high-impact failures. The SDR Illusion in Music Source Separation SDR aggregates all energy discrepancies—target scaling error, interference from other stems, and non-linear algorithmic artifacts—into one single decibel number: SDR = 10 log₁₀ (|s_target|²) / (|e_interf + e_artif + e_noise|²) Consider an AI separation model (such as Demucs or Open-Unmix) splitting a song into Vocals, Drums, Bass, and Other. The model might achieve an impressive average SDR of +12.0 dB on the vocal track. However, during a 10-second drum crescendo, sharp snare transients leak directly into the vocals at -11.6 dB. To a mixing engineer, music producer, or end listener, that vocal stem is unusable. Yet the global SDR score barely registers the localized error because it is averaged over the entire 4-minute duration of the song. Why Global SDR Fails the Mixing Engineer: Vocal Track Energy: [====================================] Global SDR: +12 dB (Pass!) Localized Bleed: [ ████ ] -11.6 dB Snare Bleed in Bar 32! ^ Global SDR hides this 10-second localized bleed that ruins the vocal mix.
The WER Blindness in Conversational Voice AI
Similarly, evaluating conversational speech models solely on Word Error Rate measures lexical transcription while ignoring temporal conversational mechanics:
• A voice assistant can transcribe every spoken word accurately (0% WER) while taking 1.5 seconds to detect that the user finished speaking. In human dialogue, a 1500 ms delay feels like an unnatural, broken interaction.
• When two speakers talk simultaneously (conversational overlap), standard ASR either drops the secondary speaker entirely or concatenates their words into a jumbled, out-of-order transcript.
Speaker A: "Are you free today?" -----------+
+--> Overlap Zone: Standard ASR collapses text
Speaker B: "Yeah, at 2 PM!" ----+
^ turnscope measures: 280ms overlap duration, +0.277s handover latency
—————————————————————————————————————————————
- What a Reproducible Diagnostic Benchmark Requires To provide actionable feedback to ML engineers, an evaluation benchmark must possess five foundational properties: +-------------------------------------------------------------------+ | REPRODUCIBLE DIAGNOSTIC BENCHMARK | +-------------------------------------------------------------------+ | | | | | v v v v v +-------------+ +-------------+ +-------------+ +----------+ +-------------+ | 1. Known | | 2. Frozen | | 3. Single- | | 4. Un- | | 5. Com- | | Ground Truth| | Protocol | | Run Test | | certainty| | ponent Diag | | by Design | | in Git | | (Unseen) | | Bounds | | (Bleed/Turn)| +-------------+ +-------------+ +-------------+ +----------+ +-------------+ | | +--> mixforge: Calibrated Bleed Synthesis | +--> stemcheck: Bleed Matrix +--> turnscope: Timing Events
Benchmark Property What It Enforces in Engineering Practice Catastrophic Failure Without It
Known Truth Ground truth is created deterministically by construction (mixforge), not predicted by the model under test. Circular Validation: The testing tool agrees with its own algorithmic biases.
Frozen Protocol Test parameters, thresholds, and data splits are committed and tagged in Git before running inference. Hypothesis Creep: Thresholds are tuned post-hoc to flatter the model.
Single-Run Test The evaluation is executed once on un-tuned holdout data. Cherry-Picking: Best-of-five runs reported as standard performance.
Uncertainty Bounds All performance figures include empirical confidence intervals. Deceptive Precision: 100% on 5 files presented as general truth.
Component Diagnosis Identifies which instrument leaks into which stem and where on the timeline. Un-Actionable Failure: Engineers know the model is bad, but not how to fix it.
—————————————————————————————————————————————
- The 3 Specialized Tools in the AudioEvaluation Suite The ComposerMind AudioEvaluation Suite provides three modular, CPU-optimized Python libraries available on PyPI: +-------------------------------------------------------------------------------------------------+ | AUDIOEVALUATION TOOL ECOSYSTEM | +-------------------------------+---------------------------------+-------------------------------+ | mixforge-audio (v0.1.0) | stemcheck (v0.1.0) | turnscope (v0.1.0) | +-------------------------------+---------------------------------+-------------------------------+ | * Deterministic multi-track | * Directional bleed matrix | * Full-duplex call timing | | * Calibrated bleed (-40 dB) | * Least-squares STFT projection | * Millisecond handover error | | * Low-pass filtered leak | * Chance-floor calibration | * Agent interruptions | | * Headroom scaling (anti-clip)| * Linear mix reconstruction | * Customer barge-in events | | * Repeatable recipe.json | * 10-second worst clash locator | * Dead-air interval detection | +-------------------------------+---------------------------------+-------------------------------+
3.1 mixforge-audio (v0.1.0): Deterministic Mixtures with Calibrated Bleed
• Core Functionality: Generates multi-track test mixtures from clean stems with mathematically calibrated physical bleed and exact recipe tracking.
• Deterministic Oracle: Embeds controlled cross-talk (e.g., injecting exactly -10.00 dB, -20.00 dB, or -30.00 dB of drum energy into isolated vocal tracks, with optional low-pass filtering to mimic natural acoustic roll-off).
• Guaranteed Linear Sum: Ensures the generated mixture equals the exact mathematical sum of the stems (mixture = \sum stems), automatically scaling overall gain when necessary to prevent clipping.
• Formal Verification: Includes mixforge verify to check that the resulting audio bit-for-bit matches the logged recipe.json.
3.2 stemcheck (v0.1.0): Directional Source Separation Forensics
• Core Functionality: Measures directional inter-stem bleed, localization timelines, and linear reconstruction completeness.
• Two Operating Modes:
-
stemcheck score(Reference-Guided): Compares separated stems against ground-truth reference stems using a bias-corrected least-squares projection on the STFT spectrogram across frequency bins. It reports an Inter-Stem Bleed Matrix where each cell (i, j) represents the decibel energy of stem j leaking into separated stem i, with a calibrated chance floor (<floor) to eliminate random coincidences. -
stemcheck check(Reference-Free): Audits separated stems with zero references: verifies whether stems linearly sum back to the original mix (\sum \hat{S}_k ≈ M), checks for clipping, DC bias, sample rate mismatch, and identifies silent or phase-cancelled channels. Provides a shared-content hint using inter-stem coherence. 3.3turnscope(v0.1.0): Conversational Interaction Analytics • Core Functionality: Evaluates two-channel full-duplex conversational audio at millisecond resolution. • Turn Handover Timing: Measures the latency between one speaker's conversational offset and the other's onset (Δt_handover), computing 50th percentile (median) and 90th percentile response times. • Conversational Event Tracking: Identifies competitive interruptions (agent_interrupts), customer barge-ins (customer_barge_in), and mutual silence (dead_air). • Channel Bleed Detection: Measures cross-talk between the two audio channels to flag acoustic feedback or hardware bleed that could distort timing measurements. ————————————————————————————————————————————— - Empirical Case Study A: Music Source Separation To demonstrate the framework, stemcheck was evaluated against controlled synthetic mixtures and real-world multi-track stems from the MUSDB18-HQ dataset (NightOwl, Clinic A, Easy Tiger) using the industry-standard HTDemucs separator. Benchmark 1: Controlled Bleed Recovery Accuracy Calibrated drum bleed was planted into isolated vocal tracks at levels from -10.00 dB down to -40.00 dB relative to the drums, both with and without -15 dB additive background noise. stemcheck score was tasked with estimating the planted leak: Planted Bleed Level Additive Noise Floor Measured Bleed Level Measurement Error Detection Rate $-10.00 dB$ None $-10.10 dB$ $0.10 dB$ 100.0% (12/12) $-10.00 dB$ -15 dB noise $-10.08 dB$ $0.08 dB$ 100.0% (12/12) $-20.00 dB$ None $-20.10 dB$ $0.10 dB$ 100.0% (12/12) $-20.00 dB$ -15 dB noise $-20.04 dB$ $0.04 dB$ 100.0% (12/12) $-30.00 dB$ None $-30.10 dB$ $0.10 dB$ 100.0% (12/12) $-30.00 dB$ -15 dB noise $-29.85 dB$ $0.15 dB$ 100.0% (12/12) $-40.00 dB$ None $-40.10 dB$ $0.10 dB$ 100.0% (12/12) $-40.00 dB$ -15 dB noise $-38.89 dB$ $1.11 dB$ 33.3% (4/12)
Key Finding: Down to $-30.00 dB$, stemcheck estimated the planted leak level with sub-0.15 dB accuracy even in the presence of -15 dB noise. At -40.00 dB with heavy noise, the measurement safely reaches the empirical chance floor rather than outputting misleading random numbers.
Benchmark 2: False-Alarm Rejection on Pristine Stems
When tested across 9 uncorrupted studio stem pairs from MUSDB18-HQ with planted unrelated artifacts, stemcheck reported 0 false alarms out of 9 tests. With a sample size of n=9, the 95% Wilson score confidence interval is [0.0\%, 30.8\%], confirming reliable initial baseline rejection while demonstrating the necessity of empirical chance-floor calibration.
Benchmark 3: Real Model Forensics on HTDemucs
Evaluating real stems generated by HTDemucs on Clinic A revealed directional flaws that global SDR completely concealed:
• HTDemucs achieved an apparently acceptable SDR of +14.8 dB (Bass) and +8.0 dB (Vocals).
• However, stemcheck flagged a severe $-11.6 dB$ directional bleed into the 'Other' stem and triggered an automated WARN on unexplained mixture residual (-21.4 dB), identifying non-linear phase cancellation during multi-track reconstruction.
HTDemucs Directional Bleed Matrix on Clinic A (stemcheck score):
+--------------+--------------+--------------+--------------+--------------+
| Output Stem | Drums Bleed | Bass Bleed | Vocals Bleed | Other Bleed |
+--------------+--------------+--------------+--------------+--------------+
| Vocals | <floor | <floor | -- | -24.2 dB |
| Drums | -- | <floor | <floor | -28.5 dB |
| Bass | -32.1 dB | -- | <floor | <floor |
| Other | -11.6 dB* | -18.4 dB | -14.2 dB | -- |
+--------------+--------------+--------------+--------------+--------------+
- Critical failure: Drums heavily bleed into 'Other' stem despite high global SDR!
—————————————————————————————————————————————
- Empirical Case Study B: Conversational Speech AI Evaluating conversational speech systems requires analyzing the exact moments where turn boundaries occur. Experiments were conducted across full-duplex conversations containing 449 human-labeled speaker handovers from the MagicHub conversational corpus. Turn Handover & Latency Benchmark Results Evaluation Metric
turnscope(Ours) Energy-Threshold Baseline Relative Improvement Speech Activity F1 Score 0.897 0.700 +28.1% F1 Improvement Precision 0.944 0.965 High precision preserved Recall 0.856 0.600 +42.7% Recall Increase Median Handover Error 0.277 seconds 0.410 seconds 32.4% Error Reduction P90 Latency Error (Worst Case) 0.440 seconds 1.339 seconds 67.1% Error Reduction Handovers Unmatched in Audio 126 / 449 207 / 449 81 More Handovers Captured
Key Conversational Analytics Findings:
- Millisecond-Accurate Handover Timing: turnscope localized conversational turn handovers with a median error of only 0.277 seconds, outperforming baseline energy detectors by 32.4\%.
- Robustness to Rapid Cross-Talk & Overlap: In challenging multi-speaker overlapping segments (the 90th percentile), turnscope constrained handover error to 0.440s, whereas conventional energy thresholding drifted to 1.339s (a 67.1\% reduction in worst-case timing error).
- Capture of Low-Intensity Speech: By tracking spectral continuity rather than raw volume, turnscope successfully captured 81 true speaker handovers that energy thresholders completely dropped. The 126 handovers not matched in audio reflect coarse human transcript boundaries that subsumed conversational pauses. —————————————————————————————————————————————
- How to Run the Diagnostic Suite Installation via PyPI Install the three evaluation packages from PyPI with pinned releases: pip install mixforge-audio==0.1.0 \ stemcheck==0.1.0 \ turnscope==0.1.0
Music Source Separation Diagnostic Workflow
1. Synthesize a golden test mix with known -26 dB drum bleed and low-pass filter
mixforge make ./stems/ --out ./golden_mix/ --bleed "drums>vocals:-26@4000" --seed 42
2. Verify mix integrity (sum of stems equals mixture)
mixforge verify ./golden_mix/
3. Score separated stems against ground truth to produce the Bleed Matrix
stemcheck score ./true_stems/ ./separated_stems/ --out ./reports/bleed_report/
4. Reference-free sanity check on separated stems against the original mix
stemcheck check song.wav ./separated_stems/ --out ./reports/sanity_report/
Conversational Turn & Overlap Analytics Workflow
Analyze full-duplex conversational audio for turn latency, interruptions, and overlap
turnscope analyze conversation_ch1_ch2.wav --json ./reports/conversational_analytics.json
Programmatic Python API for Model CI/CD
Integrate diagnostic metrics directly into automated PyTorch test suites:
from pathlib import Path
from stemcheck.audio import find_stem_files, read_audio
from stemcheck.score import score_stems
from turnscope.analyze import analyse_file
def test_source_separation_model(true_dir: Path, separated_dir: Path):
"""Automated CI test verifying that inter-stem bleed remains below -25 dB."""
refs = {name: read_audio(p) for name, p in find_stem_files(true_dir).items()}
ests = {name: read_audio(p) for name, p in find_stem_files(separated_dir).items()}
result = score_stems(refs, ests, max_bleed_db=-25.0) # Assert overall pass verdict against bleed threshold assert result.verdict() == "PASS", f"Inter-stem bleed threshold exceeded: {result.notes}" # Check specific directional bleed from drums into vocals vocals_score = next((s for s in result.stems if s.name.lower() == "vocals"), None) if vocals_score: drum_bleed = next((b for b in vocals_score.bleed if b.source.lower() == "drums"), None) if drum_bleed and drum_bleed.detected and drum_bleed.vs_target_db is not None: assert drum_bleed.vs_target_db < -25.0, f"Unacceptable drum bleed in vocals: {drum_bleed.vs_target_db:.1f} dB" def test_conversational_latency(call_audio_path: Path):
"""Automated CI test verifying agent handover latency remains under 400 ms."""
analysis = analyse_file(call_audio_path, agent_channel=0)
median_latency = analysis.pct(50)
assert median_latency is not None and median_latency < 0.400, (
f"Agent handover latency excessive: {median_latency}s"
)
—————————————————————————————————————————————
Practical Action Guide: Remediating Benchmark Failures
When diagnostic benchmarks flag a failure, the output points directly to the underlying engineering remedy:
+-------------------------------------------------------------------------+
| DIAGNOSTIC FAILURE REMEDIATION |
+-------------------------------------------------------------------------+
| | |
v v v
[ Directional Bleed ] [ Unexplained Residual ] [ Handover Latency ]
| | |
v v v
Penalize cross-stem loss Fix STFT phase mismatch Retune VAD end-of-turn
(Weight L_bleed in training) (Enforce linear sum) hangover thresholdsDirectional Bleed Detected (
stemcheck score): If drums leak into vocals, augment training loss with an explicit inter-stem interference penalty: L_total = L_target + λ \sum_{j ≠ i} |\hat{s}_i \odot s_j|^2.High Unexplained Residual (
stemcheck check): If unexplained mix is ≥ -21 dB, check for phase cancellation between stems during downmixing, or clipping introduced by independent post-processing limiters.Elevated Handover Latency (
turnscope analyze): If median response time exceeds 400 ms, shorten the voice activity detector's hangover duration (t_{hangover} < 300 ms) to ensure the endpoint detector releases promptly upon speaker silence.
—————————————————————————————————————————————Limitations and Failure Cases
To ensure rigorous usage, practitioners should keep in mind the operational boundaries of each diagnostic tool:Synthetic Bleed vs. Real Microphones (
mixforge-audio): MixForge injects linear, optionally low-passed gain-scaled bleed. Real-world acoustic spill includes complex room reflections, phase dispersion, and distance delays that this simplified model does not reproduce.Steady-State Bleed Assumption (
stemcheck): StemCheck measures stationary filter-like leakage. In dynamic soft-mask separators (such as Conv-TasNet), leaks that fluctuate rapidly from frame to frame are categorized under "unexplained residual" rather than directional bleed, causing bleed readings to read a few dB low.Channel Isolation Requirement (
turnscope): TurnScope requires distinct multi-channel recording (e.g., agent on left channel, customer on right channel). It cannot separate turn boundaries on pre-mixed mono audio without an upstream speaker diarization model.
—————————————————————————————————————————————Lessons from Failed Benchmark Designs
Building rigorous diagnostic benchmarks revealed critical methodological traps that audio ML researchers frequently encounter:Circular Ground Truth: Early validation attempts defined "clean speech" using the model's own Voice Activity Detector. The model achieved a 99.9% score because it was agreeing with its own biases. Independent human labeling is strictly mandatory.
Mathematically Unreachable Error Bounds: Demanding a false-alarm confidence upper bound of < 2.5\% requires at least 150 independent trials. Running tests on 9 or 20 files cannot mathematically guarantee low error bounds without wide confidence intervals.
Pre-Registration Discipline: When evaluating models on real recordings, pass criteria must be frozen in Git tags before running the test. Tuning thresholds after inspecting model predictions invalidates the scientific integrity of the benchmark.
—————————————————————————————————————————————Conclusion: Diagnostic Transparency for Audio AI
Moving audio and speech AI from academic prototypes to mission-critical production systems requires evaluation tools that explain where, why, and by how much a model fails.
By replacing scalar SDR and WER numbers with directional bleed matrices, linear reconstruction checks, and millisecond-accurate turn-taking analytics, the ComposerMind AudioEvaluation Suite provides researchers with the diagnostic visibility required to build robust, professional-grade audio models.
—————————————————————————————————————————————
Citation & Ecosystem Resources
• PyPI Distribution: pip install mixforge-audio stemcheck turnscope
• GitHub Repository: ComposerMind-AudioEvaluation-Suite
• Zenodo Archive: ComposerMind AudioEvaluation Suite (CERN)
• Author Contact: Boobalan Arjunan, Lead Audio AI Architect & Researcher, ComposerMind Labs, Chennai,
Role: Lead Audio AI Architect & Researcher, ComposerMind Labs
Ecosystem: ComposerMind AudioEvaluation Suite (mixforge-audio, stemcheck, turnscope)
GitHub Monorepo: github.com/BoobalanArjunan/ComposerMind-AudioEvaluation-Suite
Zenodo DOI: Zenodo Archive Record
—————————————————————————————————————————————
