Many contemporary models achieve high classification accuracy scores on the RVL-CDIP classification benchmark (e.g., [1]}, [2]}, [3]}, [4]}).
Despite this, there has been limited analysis of these models' ability to handle out-of-distribution inputs.
In this paper, we address this gap by creating a new out-of-distribution (OOD) test set intended to be used in conjunction with RVL-CDIP.
We first identify two types of OOD inputs: (1) in-domain documents that fall within one of the 16 RVL-CDIP categories, yet were sampled from a different distribution than RVL-CDIP (for example, born-digital documents from the 2010s and 2020s, and those not from the tobacco industry); and (2) out-of-domain documents that do not fall within one of the 16 RVL-CDIP categories.
Then, we evaluate several document classifiers trained on RVL-CDIP and tested on our new OOD data.
We find that model performance on both types of OOD data is substantially lower than on the in-distribution RVL-CDIP test set, indicating that while models may perform well on RVL-CDIP, they (1) struggle to generalize to other distributions and (2) struggle on the task of out-of-domain document classification.
Our hope is that with our new OOD data, researchers investigating document classification will also begin to include OOD performance as part of their analyses.
<FIGURE>