Distribution shift evaluation benchmarks are crafted to measure a model's ability to generalize to "unseen" inputs, as well as to measure how robust a model is to perturbations and/or corruptions (e.g., noise).
Here, "unseen" typically means data that is different from the training distribution, yet still belongs to the label-set of the training dataset.
For image classification, the "seen" training distribution may be photographic pictures of images but the "unseen" data distribution may be different "views" or modalities of these images, like hand-drawn sketches or clip art representations of the image categories (e.g., DomainNet [1]} and Office-Home [2]}).
