InBiaseD: Inductive Bias Distillation to Improve Generalization and Robustness through Shape-awareness
Humans rely less on spurious correlations and trivial cues, such as texture,
compared to deep neural networks which lead to better generalization and
robustness. It can be attributed to the prior knowledge or the high-level
cognitive inductive bias present in the brain. Therefore, introducing
meaningful inductive bias to neural networks can help learn more generic and
high-level representations and alleviate some of the shortcomings. We propose
InBiaseD to distill inductive bias and bring shape-awareness to the neural
networks. Our method includes a bias alignment objective that enforces the
networks to learn more generic representations that are less vulnerable to
unintended cues in the data which results in improved generalization
performance. InBiaseD is less susceptible to shortcut learning and also
exhibits lower texture bias. The better representations also aid in improving
robustness to adversarial attacks and we hence plugin InBiaseD seamlessly into
the existing adversarial training schemes to show a better trade-off between
generalization and robustness.