Out-of-scope or out-of-domain evaluation benchmarks are used to measure a model's ability to differentiate between in-domain and out-of-domain inputs.
In this setting, a model is typically trained on the in-domain data, and then evaluated on both in- and out-of-domain data (where ideally the model can correctly classify in-domain data with high accuracy while also being able to discriminate between in- and out-of-domain data, which could be done using a confidence score).
Out-of-domain benchmarks include those for text classification (e.g., intent classification in dialog systems [1]}) and image classification and object detection in the presence of natural adversarial inputs that mislead classifiers (e.g., ImageNet-O [2]} and its COCO counterpart [3]}).
In this paper, we present new out-of-domain evaluation data for the RVL-CDIP benchmark, which we call RVL-CDIP-O.
