Debunking Generalization Error or: How I Learned to Stop Worrying and
  Love My Training Set
  We aim to determine some physical properties of distant galaxies (for
example, stellar mass, star formation history, or chemical enrichment history)
from their observed spectra, using supervised machine learning methods. We know
that different astrophysical processes leave their imprint in various regions
of the spectra with characteristic signatures. Unfortunately, identifying a
training set for this problem is very hard, because labels are not readily
available - we have no way of knowing the true history of how galaxies have
formed. One possible approach to this problem is to train machine learning
models on state-of-the-art cosmological simulations. However, when algorithms
are trained on the simulations, it is unclear how well they will perform once
applied to real data. In this paper, we attempt to model the generalization
error as a function of an appropriate measure of distance between the source
domain and the application domain. Our goal is to obtain a reliable estimate of
how a model trained on simulations might behave on data.
