AI & ML
Formalizing the Intrinsic Dimension of Objective Landscapes by a Generalized Notion of Co-dimension
arXiv stat.ML0 views
arXiv:2610.39674v2 Announce Type: replace Abstract: In this work, we formalize the concept of intrinsic dimension of objective landscapes introduced in Li et al. (2018), which measures the structural complexity of learning problems by training along randomly generated subspaces. We define the intrinsic dimension of a set $\Theta\subset\mathbb{R}^p$ as the minimal dimension $d_{\mathrm{int}}(\Theta)$ such that a random plane of dimension $d_{\mathrm{int}}(\Theta)$ in $\mathbb{R}^p$ intersects $\Theta$ with probability one. We show that it is a generalized notion of co-dimension that depends on the global geometry of $\Theta$. We study mathematical properties of the intrinsic dimension and develop techniques to estimate the intrinsic dimension of sets $\Theta$ that arise in learning problems. More specifically, we derive upper and/or lower bounds where $\Theta$ is the set of optimal parameters in student-teacher settings with shallow and deep ReLU networks, including finite and infinite sample cases, and with and without regularization. We discuss how our theory leads to novel types of generalization bounds, thereby showing that the intrinsic dimension is a measure of learning complexity. Our results connect techniques from seemingly different areas of machine learning research such as classification of symmetries of ReLU networks and stochastic geometry.
Source:https://arxiv.org/abs/2610.39674PreviousBatched Stochastic Linear Bandits with 1-Bit Communication ConstraintsNextMultiply Robust Causal Mediation Analysis with Continuous Treatments
Related posts

Sources: Nvidia made two separate introductions in 2022 connecting Alibaba with alleged Chinese spy Alice Huang, actively helping Megaspeed land Alibaba (The Information)
0Techmeme
Show HN: I cut an LLM's screen-watching cost to ~138 tokens per step
0Hacker News New
Codeberg's Domain Was Suspended
0Hacker News New
Gemini AI agents repeatedly bypassed safeguards to attack real website
0Hacker News New
A new AI agent builder. No-code visual builder with customer support platform
0Hacker News New
Jacob Tsimerman on AI can supercharge mathematics [video]
0Hacker News New