AI & ML
Machine Hearing: Can Audio LLMs Generalize Beyond Human Auditory Perception?
arXiv cs.SD0 views
arXiv:2611.10693v2 Announce Type: replace Abstract: Multimodal LLMs have made remarkable progress in perceiving the physical world, while their perceptual scope remains largely human-centric, focusing on what humans can directly see or hear. This progress raises the prospect of perception beyond human limits, such as extending hearing to sounds that are inaudible to humans. More broadly, the physical world is full of waves, vibrations, and oscillations, only a small fraction of which are directly audible to humans. This invites the extension of multimodal LLMs toward a broader form of machine hearing, which remains largely underexplored. To investigate this direction, we introduce the Machine Hearing Benchmark (MHB), comprising 6,128 instances across 13 tasks and seven domains. MHB spans highly heterogeneous input regimes, with sampling rates from approximately $10^{-5}$ to $10^6$ Hz, sequence lengths from $10^1$ to $10^6$ samples, and both single- and multi-channel inputs. Evaluations of current open-source and proprietary Audio LLMs (ALLMs) reveal limited zero-shot transfer to machine hearing. Supervised adaptation on machine-hearing data alone does not close this gap, suggesting that the challenge extends beyond data exposure to a systematic mismatch between existing audio front ends and machine-hearing inputs. To address this mismatch, we introduce MachEar, a lightweight encoder extension that complements the pretrained Audio LLM for machine-hearing inputs. The results show that MachEar equips a pretrained Audio LLM with stronger machine-hearing capabilities while largely preserving its existing audio understanding performance.
Source:https://arxiv.org/abs/2611.10693PreviousA Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream TransferNextNinada: Sub-Clause Split-Graph VITS Streaming and Capability-Routed 24 kHz Speech Synthesis Across Twelve Scripts
Related posts
Voice layer for any website – does it worth?
0Hacker News New
Brazil's AI Development Strategy Needs an Edge Layer – TechPolicy.Press
0Hacker News New
Generative AI and the temporary upskilling of knowledge workers
0Hacker News New
Anthropic Unveils Claude Whiteboards, Claude Sheets, and Claude Data
0Hacker News New
AI Cracks a Quantum Computing Bottleneck
0Hacker News New
Jack and Jill Cut Candidate Screening Costs by 88% with Jev
0Hacker News New