After years of evaluation of popular supervised architectures on the test set of CIFAR-10 [1]}, modern models may overfit it through their hyperparameter tuning and structural choices. CIFAR-10.1 [2]} was collected to verify the performances of these models on a truly independent sample from the training distribution. The authors note a consistent drop in accuracy across models and suggest it could be due to a distributional shift, though they could not demonstrate it. Recent work [3]} leveraged the two-sample test to provide strong evidence of distributional shifts between the test sets of CIFAR-10 and CIFAR-10.1. We run MMD-CC and MMD two-sample tests for 100 different samplings of \(S_{\mathbb {P}}^{(1)},S_{\mathbb {P}}^{(2)},S_{\mathbb {Q}}\) , using every time \(n_{perm}=500\) , and rejecting \(\mathfrak {h}_0\)  when the obtained \(p\) -value is below the threshold \(\alpha =0.05\) . We also report results using cosine similarity applied to the features of supervised models as a comparative baseline. We report the results in Table REF  for a range of sample sizes. We compare the results to three competitive methods reported in [3]}: Mean embedding (ME) [5]}, [6]}, MMD-D [3]}, and C2ST-L [8]}. Finally, we show in Figure REF  the ROC curves of the proposed model for different sample sizes.
<FIGURE>