We now investigate our certificates in light of shifts to distribution of covariate and consider the scenario described in Sec. REF .
In this experiment we use the binary Colored MNIST dataset [1]}, which is constructed from the MNIST dataset by colouring the digits 0-4 in green and 5-9 in red for the training set, while flipping the colouring in the test set. The classifier is then trained to classify the digits into the two groups \(\lbrace 0,1,2,3,4\rbrace \)  and \(\lbrace 5,6,7,8,9\rbrace \) . In this setting, the classifier learns to perfectly distinguish the two classes in the training set, but fails on the testing set since the color is a stronger predictor than the shape of the digits.
To investigate the space between these two extreme cases, we generate mixture distributions between training and test distribution, in the following way.
We set \(P\)  to be the training distribution and \(Q\)  the testing distribution (containing digits with flipped colors).
Guided by a mixing parameter \(\gamma \) , we mix \(P\)  and \(Q\)  so as to get the mixture distribution \(\Pi _\gamma := \gamma \cdot P + (1-\gamma )\cdot Q\) .
Since \(P\)  and \(Q\)  have disjoint support, we compute the Hellinger distance between \(P\)  and \(\Pi _\gamma \)  as \(H(P,\,\Pi _\gamma ) = \sqrt{1 - \sqrt{\gamma }}\)  as shown in Appendix .
Figure REF  illustrates our robustness certificates for the 0-1 loss and the AUC Score, as well as the empirical losses \(\mathbb {E}_{\Pi _\gamma }[\ell (X, \,Y)]\)  for different values of the mixture parameter \(\gamma \) . An analogous figure for the JSD loss is given in Appendix .
We see that our technique provides quite tight
certificates, in particular for the classification error.
