AI & ML
Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers
arXiv stat.ML0 views
arXiv:2605.07297v2 Announce Type: replace Abstract: Understanding why trained Transformers generalize well is a fundamental problem in modern machine learning theory, and complexity-based generalization bounds provide a principled way to study this question. While existing norm-based bounds for Transformers remove the explicit polynomial dependence on the hidden dimension, they typically impose fixed norm constraints specified a priori and can exhibit unfavorable exponential dependence on depth. In this paper, we derive spectrum-adaptive post hoc generalization bounds for multi-layer Transformers. Under layerwise spectral norm control, the bounds are expressed in terms of layerwise Schatten quantities of the query-key, value, and feedforward weight matrices. Since the Schatten indices need not be fixed a priori and can instead be selected after training, separately for each matrix type and layer, the bounds adaptively trade off spectral complexity against the dimension- and depth-dependent factors according to the learned singular-value profiles. Empirical comparisons of BERT-adapted proxies for the leading complexity factors suggest that the proxies induced by our bounds grow more slowly with depth and hidden dimension than the corresponding norm-based proxies. Overall, our results provide a complexity-based perspective on how the spectral structure of trained Transformers is reflected in generalization analyses.
Source:https://arxiv.org/abs/2605.07297PreviousSparse Network Inference under Imperfect Detection and its Application to Ecological NetworksNextMeasuring and Decomposing Mode Separation via the Canonical Diffusion
Related posts
Embedding a Browser Game in an iframe: Keyboard Focus, Silent Failures, and a 60-Second Rule
0dev.to
10x quote gap: Class-A surfaces and T1 mold support vs render-only
0dev.to

A Bare LCD Panel Is Not a Display: What Actually Changes When You Move from Panel to Module
0dev.to
Is semantic chunking actually better than rule-based splitting for RAG retrieval?
0dev.to
Firefox RSS Reader: Get News Headlines on Your New Tab Without a Subscription
0dev.to

A slow WordPress admin: capture one request before changing plugins
0dev.to