AI & ML
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
arXiv stat.ML0 views
arXiv:2605.07297v2 Announce Type: replace Abstract: Understanding why trained Transformers generalize well is a fundamental problem in modern machine learning theory, and complexity-based generalization bounds provide a principled way to study this question. While existing norm-based bounds for Transformers remove the explicit polynomial dependence on the hidden dimension, they typically impose fixed norm constraints specified a priori and can exhibit unfavorable exponential dependence on depth. In this paper, we derive spectrum-adaptive post hoc generalization bounds for multi-layer Transformers. Under layerwise spectral norm control, the bounds are expressed in terms of layerwise Schatten quantities of the query-key, value, and feedforward weight matrices. Since the Schatten indices need not be fixed a priori and can instead be selected after training, separately for each matrix type and layer, the bounds adaptively trade off spectral complexity against the dimension- and depth-dependent factors according to the learned singular-value profiles. Empirical comparisons of BERT-adapted proxies for the leading complexity factors suggest that the proxies induced by our bounds grow more slowly with depth and hidden dimension than the corresponding norm-based proxies. Overall, our results provide a complexity-based perspective on how the spectral structure of trained Transformers is reflected in generalization analyses.
Source:https://arxiv.org/abs/2605.07297PreviousSparse Network Inference under Imperfect Detection and its Application to Ecological NetworksNextMeasuring and Decomposing Mode Separation via the Canonical Diffusion
Related posts
Elastic 8.13: GA of Amazon Bedrock in the Elastic AI Assistant for Observability
0Elastic Blog

AMD says it will outsell Intel for the first time in its history as Epyc and Instinct sales soar on AI demand
0TechRadar

SpaceX says it could build more data center compute than Google's entire lifetime capacity in just 1.5 years — and analysts think they might be able to do it
0TechRadar
Three read-only PowerShell exporters for Microsoft 365 and Windows Server admins
0dev.to
Anthropic reportedly plans to move forward with IPO plans
0TechCrunch
Handling Multi-Currency Rounding in a Cross-Border Store (and Why Refund Tags Matter)
0dev.to