- Titel
- Noise and Information in Deep Learning: A Statistical Physics Perspective on Learning and Neural Representations
- Zitierfähige Url:
- https://nbn-resolving.org/urn:nbn:de:bsz:15-qucosa2-1034476
- Datum der Einreichung
- 16.10.2025
- Datum der Verteidigung
- 03.03.2026
- Abstract (EN)
- The adoption of deep neural networks under the term of artificial intelligence in numerous real-world applications raises concerns about their robustness and interpretability. Since these models consist of large weight matrices, random matrix theory (RMT) provides a natural framework for analysis. In this thesis, we study how information is stored in the model’s weight matrices by comparing its spectra to those of randomly initialized networks, where RMT makes precise predictions. We find that deviations from these predictions in trained models indicate learned information. For image recognition tasks, such deviations grow when labels are corrupted, reflecting increased memorization demands. Removing singular values confirms that non-random ones carry essential information, while discarding noise-associated values can improve generalization when the network was trained with label noise. Extending this analysis to transformers, we find spectral deviations for both large and small singular values in non-square matrices. The associated singular vectors align with important data directions, and removing them degrades performance more than removing bulk values—contradicting the assumption that small singular values are negligible. We further analyze learning under label noise with noise-robust loss functions that prevent the learning of incorrect examples, but often lead to underfitting. By quantifying the expected backpropagation error at initialization, we propose remedies: a logit bias that aligns the network output with the regions of the loss function where correct data can be fitted, and predictive rules for optimal hyperparameters in robust losses. Both approaches resolve underfitting and achieve strong results on several benchmarks.
- Freie Schlagwörter (EN)
- Neural Network, AI, Noise-robustnesss
- Klassifikation (DDC)
- 530
- Den akademischen Grad verleihende / prüfende Institution
- Universität Leipzig, Leipzig
- Version / Begutachtungsstatus
- angenommene Version / Postprint / Autorenversion
- URN Qucosa
- urn:nbn:de:bsz:15-qucosa2-1034476
- Veröffentlichungsdatum Qucosa
- 26.03.2026
- Dokumenttyp
- Dissertation
- Sprache des Dokumentes
- Deutsch
- Lizenz / Rechtehinweis
CC BY-NC-ND 4.0