@inproceedings{
gong2023unraveling,
title={Unraveling the Complexities of Simplicity Bias: Mitigating and Amplifying Factors},
author={Xuchen Gong and Tianwen Fu},
booktitle={NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning},
year={2023},
url={https://openreview.net/forum?id=hVcGX9oOeE}
}
The success of neural networks depends on the generalization ability, while Shah et al. conclude that the inherent bias towards simplistic features, a phenomenon called *Simplicity Bias*, hurts generalization by preferring simple but noisy features to complex yet predictive ones. We aim to understand the scenarios when simplicity bias occurs more severely and the factors that help mitigate its effects. We show that many traditional insights such as increasing training size and increasing informative feature dimensions are not as effective as balancing the modes of our data distribution, distorting the simplistic features, or even searching for a good initialization. Our empirical results reveal intriguing factors of simplicity bias, and we call for future investigations to a more thorough understanding of simplicity bias and its interplay with the related fields.
Unraveling the Complexities of Simplicity Bias: Mitigating and Amplifying Factors
Xuchen Gong Tianwen Fu