Learning Robust Representations with Score Invariant Learning
Learning robust representations of data is critical for many machine learning tasks where the test distribution is different from the train distribution. Several recent approaches have proposed new principles to achieve generalizable predictors by learning robust representations from multiple training set distributions. These studies develop new algorithms that guide neural networks to learn representations that are stable across training sets, rather than spurious correlations within any given training set. We develop a new principle for learning robust representations: score invariant learning (SIL) We observe that learning a model over multiple train domains can be viewed as learning a distinct model distribution over data from each domain, with the property that all model distributions share the same parameters. We use this observation to derive an invariance metric, based on score matching, that learns robust estimators by enforcing invariance of likelihood of points across model distributions. Our experiments demonstrate SIL achieves stateof the art performance on the Colored MNIST task, as well as performance competitive with the state of the art on two domain generalization benchmarks, PACS and VLCS.