A recent comprehensive study [1]} analyzed the out-of-domain generalization and robustness of ImageNet models on several datasets (which have distribution shifts) and found that across all datasets, the accuracy of models dropped well below the expectation set by the ImageNet validation set. A few datasets tested are: ImageNet-Adversarial [2]} (contains natural adversarial images), ImageNet-R [3]} (renditions), ImageNet-Sketch [4]} (sketches), ImageNet-Real [5]} (corrected labels in original dataset), ImageNet-V2 [6]} (new test set for ImageNet benchmark), ObjectNet [7]}. Each of these datasets have the subset or same labels as the original ImageNet-1K and we use these dataset for our models benchmarking.
