AI & ML
Speed limits for information acquisition in stochastic learning dynamics
arXiv stat.ML0 views
arXiv:2609.08219v2 Announce Type: replace-cross Abstract: Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about data-generating parameters. The resulting inequality decomposes the upper bound on the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different data-generating parameters are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for understanding which properties of the data-generating process are acquired, in what order, and how rapidly they can be encoded in the trainable parameters.
Source:https://arxiv.org/abs/2609.08219PreviousDeep Barycentric Regression for Optimal Transport Map Estimation and Its Statistical OptimalityNextSymplectic Hamiltonian Direct Discontinuous Galerkin Method for Wave Propagation
Related posts
Lakewood Shooting: One Police Officer Killed, Three Wounded in Colorado
0dev.to News
OpenAI: Images 2.6, GIF e Workspace
0dev.to News
How to Plan a Healthtech Product Video Without Sounding Like a Brochure
0dev.to Design
Building a Stress Relief Toy Brand From Wholesale Sourcing to Custom Molds
0dev.to Design
Người trẻ lễ Phật trên điện thoại: trào lưu đi chùa online lan nhanh
0dev.to Discuss

My AI agent says "all fixed." 3 times out of 4, it never measured the before.
0dev.to Discuss