ER-TEST: Evaluating Explanation Regularization Methods for NLP Models

Neural language models' (NLMs') reasoning processes are notoriously hard to explain. Recently, there has been much progress in automatically generating machine rationales of NLM behavior, but less in utilizing the rationales to improve NLM behavior. For the latter, explanation regularization (ER) aims to improve NLM generalization by pushing the machine rationales to align with human rationales. Whereas prior works primarily evaluate such ER models via in-distribution (ID) generalization, ER's impact on out-of-distribution (OOD) is largely underexplored. Plus, little is understood about how ER model performance is affected by the choice of ER criteria or by the number/choice of training instances with human rationales. In light of this, we propose ER-TEST, a protocol for evaluating ER models' OOD generalization along three dimensions: (1) unseen datasets, (2) contrast set tests, and (3) functional tests. Using ER-TEST, we study three key questions: (A) Which ER criteria are most effective for the given OOD setting? (B) How is ER affected by the number/choice of training instances with human rationales? (C) Is ER effective with distantly supervised human rationales? ER-TEST enables comprehensive analysis of these questions by considering a diverse range of tasks and datasets. Through ER-TEST, we show that ER has little impact on ID performance, but can yield large gains on OOD performance w.r.t. (1)-(3). Also, we find that the best ER criterion is task-dependent, while ER can improve OOD performance even with limited and distantly-supervised human rationales.


Introduction
Neural language models (NLMs) have achieved state-of-the-art performance on a broad array of natural language processing (NLP) tasks (Devlin * Equal contribution. Figure 1: Explanation Regularization: Given an instance and a target label, we can use rationale extractors (See Section 2) to generate machine rationales from a model F. Furthermore, human rationales are collected from annotators. Explanation Regularization (ER) aligns machine rationales to human rationales with a loss term, L ER , which is then used to refine F. et al., 2018;Liu et al., 2019). Even so, NLMs' reasoning processes are notoriously opaque (Rudin, 2019;Doshi-Velez and Kim, 2017;Lipton, 2018), which has spurred significant interest in designing algorithms to automatically explain NLM behavior (Denil et al., 2014;Sundararajan et al., 2017;Camburu et al., 2018;Rajani et al., 2019;Luo et al., 2021). The majority of this work has focused on rationale extraction, which explains a NLM's output on a given task instance by highlighting the input tokens that most influenced the output (Denil et al., 2014;Sundararajan et al., 2017;Li et al., 2016;Jin et al., 2019;Lundberg and Lee, 2017).
Recently, a number of works have investigated how machine rationales produced by rationale extraction algorithms can be operationalized to improve NLM decision-making (Hase and Bansal, 2021) (See Figure 1). Almost all prior works are based on explanation regularization (ER), which aims to improve NLM generalization by regularizing the NLM to yield machine rationales that align with human rationales (Ross et al., 2017;Huang et al., 2021;Ghaeini et al., 2019;Zaidan and Eis-ner, 2008;Kennedy et al., 2020;Rieger et al., 2020;Liu and Avci, 2019). Human rationales can be created by annotating each training instance individually (Lin et al., 2020;Camburu et al., 2018;Rajani et al., 2019) or by applying task-level human priors across all training instances (Kennedy et al., 2020;Rieger et al., 2020;Ross et al., 2017;Liu and Avci, 2019).
Although prior works primarily evaluate such ER models via in-distribution (ID) generalization (Zaidan and Eisner, 2008;Lin et al., 2020;Huang et al., 2021), out-of-distribution (OOD) generalization is more crucial in many real-world settings (Chrysostomou and Aletras, 2022;Ruder, 2021), yet ER's impact on OOD generalization is largely underexplored (Ross et al., 2017;Kennedy et al., 2020). Plus, despite them being major factors in ER, little is understood about how ER model performance is affected by the choice of ER criterion or by the number/choice of training instances with human rationale supervision. In light of this, we propose ER-TEST, a protocol for evaluating ER models' OOD generalization along three dimensions: (1) unseen datasets, (2) contrast set tests, and (3) functional tests. For (1), ER-TEST assesses ER models' task performance on datasets beyond their training distribution. For (2), ER-TEST assesses ER models' sensitivity to counterfactual instances created by perturbing existing datasets. For (3), ER-TEST assesses ER models' basic linguistic capabilities (e.g., perception of word/phrase sentiment, robustness to typos) expected for the given task.
Using ER-TEST, we study three key questions: (A) Which ER criterion are most effective for the given OOD setting? (B) How is ER affected by the number/choice of training instances with human rationales? (C) Is ER effective with distantly supervised human rationales? ER-TEST enables comprehensive analysis of these questions by considering a diverse range of text classification tasks and datasets. Through ER-TEST, we show that ER has little impact on ID performance (Sec. 5.3.1, 5.4.1), but can yield large gains on OOD performance (Sec. 5.3.2,5.4.2) w.r.t. (1)-(3). Also, we find that the best ER criterion is task-dependent (Sec. 5.3), while ER can improve OOD performance even with limited human rationale supervision (Sec. 5.4). Furthermore, we show that ER can improve OOD performance using distantly supervised human rationales (Sec. 5.5).

Background
Text Classification Let F be a NLM task model for M -class text classification. In modern NLP systems, F usually has a BERT-style architecture (Devlin et al., 2018), consisting of a Transformer encoder (Vaswani et al., 2017) followed by a linear layer with softmax classifier. Let x i = [x t i ] n t=1 be the n-token input sequence (e.g., a sentence) for task instance i. For sequence classification, F predicts a class for sequence x i , so let F(x i ) ∈ R M be the logits for x i . Let y i = arg max c F(x i ) c denote F's predicted class for x i . For token classification, F predicts a class for each token x t i , so let F(x i ) ∈ R n×M be the logits for the n tokens in x i . Let y i,t = arg max c F(x i ) t,c denote F's predicted class for x t i . Let y i = [y i,t ] n t=1 collectively denote all of F's predicted token classes for x i .
Rationale Extraction Given F, x i , and y i , the goal of rationale extraction is to output machine rationale r i = [r t i ] n t=1 , such that each r t i ∈ [0, 1] is an importance score indicating how strongly token x t i influenced F to predict class y i . Let G denote a rationale extractor, such that r i = G(F, x i , y i ). G first computes raw importance scores s i ∈ R n , then normalizes s i as probabilities r i using the sigmoid function. In general, G can be a heuristic or learned function, but we focus on heuristic G in this work, since they are more common (Luo et al., 2021). Broadly, heuristic G's can either be a gradient-based, that assign importance scores based on gradient changes in F (Sundararajan et al., 2017;Sanyal and Ren, 2021;Shrikumar et al., 2017), sampling-based, that assign important scores based on the neighbours/context of a given token (Zeiler and Fergus, 2013;Jin et al., 2019), or attentionbased, that use the attention-scores or a function of them to assign importance scores. (Ding and Koehn, 2021) Explanation Regularization (ER) However, G can also be used to compute machine rationales w.r.t. other classes besides y i , e.g., target classẏ i . Letr i denote the machine rationale for x i w.r.t.ẏ i . Givenr i obtained via G and F, many works have explored ER, in which F is regularized such thatr i aligns with human rationaleṙ i (Zaidan and Eisner, 2008;Lin et al., 2020;Rieger et al., 2020;Ross et al., 2017). Typically,ṙ i is a binary vector, where ones and zeros indicate positive (important) and negative (unimportant) tokens, respectively. ER's inductive bias pushes F to solve the task in a way that follows the human reasoning process given bẏ r i , which ideally provides denser learning signal for improving F's generalization.
We formalize the ER loss as: L ER = Φ(r i ,ṙ i ), where Φ is an ER criterion measuring the alignment betweenr i andṙ i . Thus, the full learning objective is: L = L task + λ ER L ER , where L task is the task loss (e.g., cross-entropy loss) λ ER ∈ R is the ER strength (i.e., loss weight) for L ER . Let γ ER > 0 be the rationale scaling factor, used to scaleŝ i prior to sigmoid normalization. If the magnitudes of theŝ i scores are lower, then ther i scores will be closer to 0.5 (i.e., lower confidence). However, scalingŝ i by γ ER > 1 will increase the magnitude ofŝ i , yieldinĝ r i scores closer to 0 or 1 (i.e., higher confidence). Though there are many possible choices for Φ, it is presently unclear how different Φ impact training and when certain Φ should be preferred. This limits our ability to use ER in real-world settings.

ER-TEST
Existing works primarily evaluate ER models via ID generalization (Zaidan and Eisner, 2008;Lin et al., 2020;Huang et al., 2021), though a small number of works have done auxiliary evaluations of OOD generalization (Ross et al., 2017;Kennedy et al., 2020;Rieger et al., 2020). However, these OOD evaluations have been relatively small-scale, only covering a narrow range of OOD generalization aspects, ER criteria, training settings, tasks, and datasets. As a result, little is understood about ER's impact on OOD generalization. To address this gap, we propose ER-TEST, a unified benchmark for evaluating ER models' OOD generalization along three dimensions: (1) unseen datasets; (2) contrast set tests; and (3) functional tests.

ID Generalization
While ER-TEST's main focus is on evaluating OOD generalization, ER-TEST also considers ID generalization as a baseline evaluation. Let are the target classes, and N is the number of instances (x i ,ẏ i ) in D. We call D the ID dataset. Assume D can be partitioned into train set D train , dev set D dev , and test set D test , where D test is an ID test set for D. After using ER to train F on D train , we measure F's task performance on the ID test set D test . Note that this is a standard protocol used by existing works to evaluate ER models (Zaidan and Eisner, 2008;Rieger et al., 2020;Liu and Avci, 2019;Ross et al., 2017;Huang et al., 2021;Ghaeini et al., 2019;Kennedy et al., 2020).

OOD Generalization
To assess F's generalization ability when using ER, we consider various OOD settings. Given D, let D test denote an OOD test set, with a different distribution from D. While F is expected to perform well on D test (ID), F should also perform well oñ D test (OOD). For each dimension of OOD generalization, we obtainD test in a different manner.

Unseen Datasets
First, we evaluate OOD generalization w.r.t. unseen datasets. Besides D, suppose we also have a set of datasets {D (1) ,D (2) , ...} of the same task as D. Each of these datasetsD (i) has its own train/dev/test sets and a distribution shift from D. After using ER to train F on D train , we measure F's task performance on each OOD test setD In other words,D (i) test is obtained by simply taking the test set of existing OOD datasetD (i) . This evaluation is designed to assess whether ER helps F learn general (i.e., task-level) knowledge representations that can (zero-shot) transfer across datasets.

Contrast Set Tests
Second, we evaluate OOD generalization under meaningful dataset perturbations. Annotation artifacts (Gururangan et al., 2018) are gaps present in a dataset that can lead to misleading interpretations of a model's performance on that dataset. To mitigate this, we evaluate F on contrast sets (Gardner et al., 2020), which are (mostly) labelchanging small perturbations on instances to understand the true local boundary of the dataset. Essentially, they help us understand if F has learnt any dataset-specific shortcuts. GivenD p is applied to that instance, where p denotes the kind of perturbation taking effect, and it often changes the target label for that instance. For example, p can signify semantic (e.g., changing tall to short), numeral (e.g., changing one dog to three dogs), or entities (e.g., changing dogs to cats). Each perturbation type is specific to the dataset it is being created for, so that instance labels are changed in a meaningful manner. The resulting set of instances test (j))∀j, p are termed as a contrast set for that dataset. Based on the way they are created, contrast sets are a property of the dataset, and are not created to explicitly challenge F (unlike adversarial examples (Gao and Oates, 2019)).

Functional Tests
Third, we evaluate OOD generalization w.r.t. functional tests (Ribeiro et al., 2020;Li et al., 2020). Unlike contrast sets which are designed to test artifacts present in a dataset, functional tests are used to provide 'zoomed-in' insights about specific linguistic capabilities (like changes in the vocabulary, adding negations to instances, etc). Furthermore, contrast sets are created by perturbing a reference real-world dataset, whereas, functional tests evaluate specific capabilities with the help of templategenerated synthetic instances.
If ER consistently improves F's performance on such tests, then we can have higher confidence that ER is a useful inductive bias for OOD generalization for that given capability Across all tasks, ER-TEST considers four categories of stress tests, which are adopted from CheckList (Ribeiro et al., 2020). Each test is described below.
Vocabulary Tests Vocabulary tests are used to evaluate F's capability to address changes in the vocabulary of the text, and is particularly diverse w.r.t the parts-of-speech it caters to. For example, certain vocabulary tests evaluate the relationship (taxonomy) between different nouns in a sentence, whereas some swap the modifiers or the verbs present in a sentence in a meaningful manner based on the task at hand, to capture F's targeted performance towards such changes (Ribeiro et al., 2020).
Robustness Tests Robustness tests evaluate F's behavior under character-level edits to words in a sentence, keeping the rest of the context same so as to not change the overall prediction. They include testing against typos as well as contractions in words, as well as addition of tokens that are irrelevant for the downstream task (like URLs or gibberish like Twitter handles). (Jones et al., 2020;Wang et al., 2020) Logic Tests Testing F's reasoning capabilities towards logical changes in a sentence is also important to evaluate its reliance on shortcut-patterns. These tests perturb sentences in a logical manner (by adding or removing negations, or purposefully inducing contradictions) that also change the target label in the same manner. (Talman and Chatzikyriakidis, 2018;McCoy et al., 2019) Entity Tests For certain tasks, named entities like numbers, locations and proper nouns are not relevant for predicted a target label, and are often a source of gender or demographic biases (Mishra et al.;Mehrabi et al., 2020). Entity tests measure F's sensitivity towards changed in named entities such that the overall context as well as the task label remains the same (Ribeiro et al., 2020).

Tasks and Datasets
To evaluate ER models, ER-TEST considers a diverse set of sequence and token classification tasks. For each, task ER-TEST provides one ID dataset (annotated with human rationales) and multiple OOD datasets. Compared to prior works, ER-TEST's task/dataset diversity enables more extensive analysis of ER model generalization.
First, we have sentiment analysis, using SST (movie reviews) ( Figure 3: Analysis Setup: We use ER-TEST to answer three RQs in this work -RQ1 addresses which ER criteria work for which task, and evaluates their improvements, RQ2 evaluates whether a particular method of selecting a fixed budget of samples for annotating human rationales yields performance improvements, and RQ3 studies ER w.r.t. another dimension of resource constraints, where we can only provide distant supervision to generate human rationales. We lay out more details in Section 4. 2020) as the ID dataset. For OOD datasets, we use Yelp (restaurant reviews) (Zhang et al., 2015), Amazon (product reviews) (McAuley and Leskovec, 2013), and Movies (movie reviews) (Zaidan and Eisner, 2008;DeYoung et al., 2019). Movies' inputs are much longer than the other three datasets'. For contrast set tests, we use an OOD contrast set for sentiment analysis released by the authors of the original paper (Gardner et al., 2020), which are created for the Movies dataset. Furthermore, for functional tests, we use an OOD test suite (flight reviews) from the CheckList (Ribeiro et al., 2020) which contains both template-based instances to test linguistic capabilities, as well as real-world data (tweets).

Distantly Supervised Human Rationales
Typically, human rationales are created by annotating each training instance individually (Lin et al., 2020;Camburu et al., 2018;Rajani et al., 2019). For each training instance, humans are asked to mark tokens that support the gold label as positive, while the remaining tokens are marked as negative.
Here, each human rationale is specifically conditioned on the input and gold label for the given instance. However, such instance-level human rationales are very expensive to obtain, given the high manual effort per instance.
Alternatively, some works have constructed distantly supervised human rationales by applying task-level human priors across all training instances (Kennedy et al., 2020;Rieger et al., 2020;Ross et al., 2017;Liu and Avci, 2019). For example, Kennedy et al. (2020) used a "blacklist" lexicon to distantly supervise human rationales for the hate speech detection task. In the past, hate speech detection models were largely oversensitive to certain group identifier words (e.g., "black", "Muslim", "gay"), almost always predicting hate speech for text containing these words. To address this, they first manually annotated a lexicon of group identifiers that should be ignored for hate speech detection. Then, for all training instances, they automatically marked only tokens belonging to the lexicon as negative (and the rest as positive). By using these human rationales for ER, they trained the NLM to be less biased w.r.t. these group identifiers. For the purpose of our study, we use the lexicons as used by (Jin et al., 2021) to generate distantly-supervised rationales for the Stormfromt (Stf) dataset (de Gibert et al., 2018). Each instance in the Stf dataset is matched to one or more lexicons by simple character-level matching, and the rationales are generated as described above. We evaluate ER methods on OOD hate speech detection datasets like HatEval (Barbieri et al., 2020) and Gab Hate Corpus (GHC) (Kennedy et al., 2018). All of the datasets contain binary labels for hateful and non-hateful content. The Stf dataset is collected from a white-supremacist forum, whereas HatEval instances are tweets and GHC instances are taken from the Gab forum.

Analysis Setup
After introducing the ER-TEST framework, we conduct a systematic study of ER through three primary research questions (described below) using ER-TEST ( Figure 3). First, we aim to study which ER criteria are effective for a given task at hand (Section 4.1). Second, we study ER from a resource constraint perspective, where only a handful of instances can have human rationales annotated. What is key here is to define how to select these instances for rationale annotation (Section 4.2). Lastly, we look at human rationales that are distantly-supervised and how ER can help improve performance when dense, instance-level human rationale annotations are not available (Section 4.3).
4.1 RQ1: Which ER criteria are most effective?
Compared to existing works, ER-TEST uses a wider range of ER criteria to evaluate ER model generalization. This provides a more comprehensive picture of ER's impact on both ID and OOD generalization. Also, this can help us understand why certain criteria work well and under what settings they work best. To demonstrate the utility of ER-TEST, we consider five representative ER criteria (i.e., choices of Φ).

Mean Absolute Error (MAE) MAE is used in
Binary Cross Entropy (BCE) BCE loss is used in Chan et al. (2021).
Huber Loss Huber loss (Huber, 1992) is a hybrid of MSE and MAE. ΦHuber(ri,ṙi) Order Loss Recall that the human rationaleṙ i labels each token as positive (one) or negative (zero). Whereas other criteria generally push positive/negative tokens' importance scores to be as high/low as possible, order loss (Huang et al., 2021) relaxes MSE to merely enforce that all positive tokens' importance scores are higher than all negative tokens' importance scores. This is especially useful ifṙ i is somewhat noisy, e.g., some positivelylabeled tokens should not really be positive.
4.2 RQ2: How is ER affected by the number/choice of train instances with human rationales?
In real-world applications, it is infeasible to obtain human rationalesṙ i for all of the instances in the training set, asṙ i requires dense annotation (Chiang and Lee, 2022;Kaushik et al., 2019). Let S be a subset of train instances for which we have human rationale annotations,ṙ S i . Therefore, the ER loss L S ER = Φ(r S i ,ṙ S i ) and the full learning objective L = L task + λ S ER L S ER , where L task is computed on the full dataset as it would normally.
In designing such a system, one needs to carefully select S that leads to highest performance gains, meanwhile maintaining resource constraints. To select relevant samples to annotate, existing methods use to active-learning based approaches (Schröder and Niekler, 2020). We use ER-TEST to compare three such approaches approaches to select S: Random Sampling Given a k, we uniformly select k% of samples from D to construct S.
Lower Confidence (LC) Sampling Given a k, we select top k% of samples ordered on the basis of the Lower Confidence criterion. (Zheng and Padmanabhan, 2002) where (x i ,ẏ i ) ∈ D train , and θ are the parameters of a model trained on D train without ER. In other words, these are the top k% examples that a model trained without ER is the least confident on.
Higher Confidence (HC) Sampling Given a k, we sample the top k% of samples ordered in the reverse order of lower confidence prioritisation as described above. In other words, these are the top k% examples that a model trained without ER is the most confident on.

RQ3: Is ER effective with distantly supervised human rationales?
In Section 4.2, we enforce constraints in the number of instances to annotate with human rationales, and use ER-TEST to compare different strategies to select such instances. However, annotating each token within each instance is also an expensive task, as described in Section 3.4. One way to overcome this issue is generate human rationales with distant supervision. Let L D be a list of lexicons curated by human annotators, specific to a given dataset D. Let l(·) be an indicator function that searches for a given lexicon list in all the tokens of an instance, and returns a binary representation of the same size as the instance with 1s in places with lexicon matches (0 otherwise). Therefore, we can obtain distantly- We can then study the effectiveness of ER methods as detailed in RQ1 (Section 4.1) in this setting.

Implementation Details
For the NLM architecture, we use BigBird-Base (Zaheer et al., 2020), in order to handle input sequences of up to 4096 tokens. For all results, we report the mean over three seeds, as well as the standard deviation. By default, we use a learning rate of 2e−5 and effective batch size of 32. For ER, there are many possible choices of rationale extractor G, but evaluating all of these choices would be prohibitive. Also, evaluating G is orthogonal to ER-TEST's goal of evaluating ER criteria Φ. Thus, as a proof of concept, we use the Input*Grad algorithm (Denil et al., 2014) as G in all experiments, given its popularity and computational efficiency (Bastings and Filippova, 2020;Luo et al., 2021). We leave investigation of other G for future work.

Intrinsic Evaluation of ER
ER in general is sensitive to certain hyperparameters for yielding meaningful training curves and actually attaining alignment between machine and human rationales. Due to a large set of tunable hyperparameters, running all configurations of ER are not feasible. Therefore, we intrinsically evaluate hyperparameter configurations by assessing the loss curves (which model alignment between machine and human rationales) w.r.t different hyperparameters values. We observe that the acceptable band of learning rates for ER is very narrow, and we use 2e−5 in all of our experiments. Furthermore, we also observe that setting λ ER = 1 and γ ER = 100 yields the most drop in the loss curves while training, so we use these hyperparameters for the rest of our experiments. We detail these experiments in Appendix A.1.
5.3 RQ1: Which ER criteria are most effective?

ID Generalization
In Table 1 (In-Distribution), we display the ID task performance results for sentiment analysis (SST), NLI (e-SNLI), and NER (CoNLL-2003). For SST, we find that all of the ER criteria yield about the same task performance as the None baseline, whereas, all ER criteria also perform similarly for NLI (yielding higher performance than None). For NER, we see more variance in task performance among ER criteria, although the variance is still quite small among the best methods (MSE, MAE, Huber). Here, MAE yields the highest task performance, while BCE yields the lowest by far. Overall, using ID task performance, it is difficult to distinguish between ER criteria and underplays its overall benefits. This motivates us to consider other evaluation metrics.

OOD Generalization Unseen Datasets
In Table 1  and compares them to a setting without ER (None). All models (with or without ER) are trained on the ID dataset and evaluated on the ID and OOD datasets without the need of machine or human rationales. Metrics displayed here (higher the better) for sentiment analysis is Accuracy and Macro F1 for NLI and NER. and correspond to cases where the ER criterion in bold are significantly similar and greater than None respectively (p < 0.05).
Meanwhile, despite performing best on SST, Order performs much worse than all other ER criteria here. For NER, MAE still performs best, while MSE and Huber are competitive. Overall, OOD task performance is much better than ID at distinguishing between ER criteria, especially showing ER's improvement over None.

Contrast Set Tests
In Table 2 (Contrast Set Analysis), we observe the drop in performance (denoted by ∆) for sentiment analysis (Movies) and NLI (MNLI) when using a contrast set designed for the given dataset. We observe that for both of the tasks, MAE leads to the least drop in performance. All of the methods apart from Order yield lower drops than None. All of them also have a higher performance on the original and contrast sets. For sentiment analysis, we observe that Order has the highest variance, and for NLI, it has the highest drop in performance. We believe that some of it can be attributed to the soft-ranking that is imposed by Order, which may be indifferent towards minor label-changing edits, that is observed by the contrast sets. Figure 4 demonstrates the failure rates on functional tests (as listed in Section 3.2.3) of our ID models trained on the sentiment analysis and NLI tasks. We also present the overall aggregated (over each individual tests within the categories mentioned) failure rates.

Functional Tests
We observe that apart from the entity-based tests, ER criteria generally have a lower failure rate than None for all of the other tests. For entity-based tests, ER criteria either perform comparably (sentiment analysis) or worse (NLI) than None. Generally, all methods perform well on robustness-based tests, as they have lower failure rates, with order loss having the least. What is important to note is the significant improvement by order loss in vocabulary-based tests than None, even though all of the methods are exposed to the same training set instances. We hypothesize that the biases induced by ER alleviates the shortcuts learnt by None. This is also validated by the overall performance on all of these stress tests, where all of the ER criteria (apart from Huber in sentiment analysis) have lower failure rates than None.  Table 1 on sentiment analysis. Furthermore, all hyperparameters are same as that detailed in Section 5.2. We also choose the best performing ER criterion for sentiment analysis from Table 1 (MAE) and base the following results on the same criterion.

ID Generalization
Consistent with the results we observed in Table  1, Table 3 shows us that there are little to no improvements in ID performance with prioritisation methods, however performance for all of them (except LC) is maintained at par with that of None and 100% sampling. This shows that doing ER on selective samples does not degrade ID performance.

OOD Generalization (Unseen Datasets)
Interestingly, we observe distinctions within various prioritisation methods as we look at OOD evaluations in  original and contrast sets. ∆ is the difference in performance of F between the contrast and original set, and lower the value, better the generalization power of F. A value farther from 0 suggests that F has learnt shortcuts specific a dataset, which are not generalizable to task that the dataset captures as a whole. outperform None. This implies that doing ER on a smaller subset of instances would instantly yield small improvements over None. As we increase the annotation budget, we observe that model performance declines as we select lower confidence samples, but is maintained or even improves over random while selecting instances with greater confidence. It is important to reiterate here, that samples are prioritized based on the confidence yielded by the None model. This implies that models in general require inductive biases on samples they are already confident on, vs. samples they are less confident on to avoid confusion. We plan to study this phenomenon in more detail for other tasks and more constrained annotation budgets as a part of our Future Work (as detailed in Section 6).
5.5 RQ3: Is ER effective with distantly supervised human rationales?

ID Generalization
For the task of hate speech detection, we train F with the Stf dataset. We report all accuracies in Table 4. As it was observed in Section 5.3.1, ER does not lead to a significant improvement in performance for the Stf test set. However, it is important to note that "blacklisting" group identifier lexicons does not lead to a drop in ID performance either. Benefits of "blacklisting" are then observed in OOD generalization.

OOD Generalization
Unseen Datasets We evaluate F on two OOD datasets, HatEval and GHC. Table 4 shows that while the improvements in HatEval are not significant, there are significant accuracy improvements for the GHC test set, which are due to the Order ER criterion.
Fairness Tests In addition to generic performance metrics like accuracy, we also measure group identifier bias (against the groups detailed by group identifier lexicons) by evaluating the False Positive Rate Difference (FPRD) as shown by (Jin et al., 2021). FPRD is computed as z |FPR z − FPR overall |, where FPR z is the false positive rate of all of the test instances mentioning group identifier z, and FPR overall is the false positive rate of all the test instance. Essentially, FPRD evaluates if F is more biased against a given group identifier z, than all of the groups. A lower FPRD value indicates less biased against the listed group identifiers by F.   ria suffer with higher bias than None, we observe that Order criterion consistently leads to the least bias, both in-distribution and out-of-distribution. Furthermore, the reduction in bias is significant when compared to None. Interestingly, Order ER criterion was initially conceived for distantlysupervised rationales (Huang et al., 2021), and the authors of the original paper also demonstrated experiments with rationales generated from lexicons where Order criterion leads to improvements. Our observations are in-line with theirs, and we also show its benefit in reducing bias in F.

A Appendix
A.1 Intrinsic Evaluation: evaluating ER's sensitivity to hyperparameters When using ER to train F, it is important to assess whether ER exhibits expected training behavior, orthogonally to task performance. If ER improves task performance, this kind of analysis can help us better understand ER's effectiveness. Conversely, if ER does not improve task performance, such analysis can help us identify the problem. Motivated by this, ER-TEST's intrinsic evaluation is based on machine-human rationale alignment, captured by the ER loss L ER = Φ(r i ,ṙ i ).
When using ER, we should generally expect the ER loss to decrease as F is trained. In practice, this may not always be the case, even when ER leads to slightly higher task performance (which is likely a mirage caused by lucky random seeds)! That is, by definition, non-decreasing ER loss signals ineffective ER usage, since the machine rationales are not becoming more similar to the human rationales. This can stem from a number of issues: e.g., poor choice of ER criteria Φ, improper ER strength λ ER , improper rationale scaling factor γ ER , noisy human rationaleṙ i , insufficient F capacity. Thus, we measure machine-human rationale alignment as the first step in diagnosing such issues.
Let ER loss curve denote a chart which plots L ER vs. the number of train epochs. For each combination of ER criteria Φ and some training configuration, we plot ER loss curves for the training set. Each component of our intrinsic evaluation varies a different hyperparameter in the training configuration: (A) ER strength λ ER ; (B) rationale scaling factor γ ER ; and (C) learning rate α. In contrast, prior works do not explore the relationship between L ER and these training variables (Huang et al., 2021;Ghaeini et al., 2019).
For intrinsic evaluation, we use ER strength λ ER = 1, rationale scaling factor γ ER = 1, and learning rate α = 2e−5, unless otherwise specified. As a proof of concept, we focus on SST here, but plan to add other datasets in future work.

A.1.1 ER Strength
Fig. 6 displays the ER loss curves for different ER strengths λ ER = [0.5, 1, 10, 100, 300], on SST using MAE. Among the λ ER values, we see that λ ER = 1 yields ER loss curves with the greatest decrease (Table 6), signaling good ER optimization.     ER-trained model, evaluated on either the dev set or test set (yielding two point sets). Fitting each point set with linear regression, we find that there is an inverse relationship between task performance and ER loss. In other words, higher machine-human rationale alignment (i.e., low ER loss) corresponds to higher task performance, which validates the usage of ER to improve generalization. Table 7 displays the slopes and R 2 scores of the lines in Fig. 9. The slope indicates the strength of the relationship between machine-human rationale alignment and task performance (lower is better), while the R 2 score indicates how accurately each line fits its corresponding data points. Among the five ER criteria, across dev and test, we find that MAE has the lowest slopes and highest R 2 scores overall, suggesting that using ER with MAE is most effective.

A.2.3 Change in Target Class Confidence
We consider ER with the MAE criterion, trained/evaluated on SST (via dev ID task performance). Fig. 10 Fig. 9 (ER Loss vs. Task Performance), using slope and R 2 score (Sec. A.2.2). Ideally, Fig. 9's lines would have low slope and high R 2 , indicating that ER helps improve task performance. We see that MAE yields the best ER results.  where FNo-ER's target class confidence is low, there is a higher percentage of instances that are predicted incorrectly/correctly without/with ER. This suggests that instances with low target class confidence are more likely to benefit from ER.
Among instances for which F No-ER 's target class confidence is low, there is a higher percentage of instances that are predicted incorrectly/correctly without/with ER (i.e., incor→cor). This suggests that, for F No-ER , instances with low target class confidence are more likely to benefit from ER (Table  8). Also, based on the T-test, target class confidence scores are significantly higher (p < 0.005) with ER than without.

A.2.4 ER Opportunity Cost
An ER-trained NLM F task, ER and a non-ER-trained NLM F task, No-ER are likely to yield different outputs given the same inputs. Let D + ER ⊆ D and D + No-ER ⊆ D denote the sets of instances predicted correctly by F task, ER and F task, No-ER , respectively. Ideally, we would have D + No-ER ⊂ D + ER . This means there is no opportunity cost in us- ing ER, as ER increases the number of correct instances without turning any previously-correct incorrect. However, this may not necessarily be the case, so we measure ER's opportunity cost as follows. Let n + ER = |D + ER \(D + ER ∩ D + No-ER )| be the number of instances predicted correctly by F task, ER , but not by F task, No-ER . Let n + No-ER = |D + No-ER \(D + No-ER ∩ D + ER )| be the number of instances predicted correctly by F task, No-ER , but not by F task, ER . Then, the opportunity cost of using ER is defined as: In practice, instead of defining o ER for all of D, we only consider test sets D test andD test . Table 9 displays the opportunity cost results for sentiment analysis. Generally, the opportunity cost results mirror the task performance results in Table  1, such that the methods with highest task performance tend to have the lowest opportunity cost. However, using opportunity cost, the variance is very high for OOD datasets, making it difficult to compare methods. In future work, we plan to modify the opportunity cost metrics to better accommodate OOD settings.

A.3 Efficient hyperparameter tuning with ER-TEST
In intrinsic evaluation (Sec. A.1), we used ER loss curves as priors for selecting three key ER hyperparameters (i.e., ER strength λ ER , rationale scaling factor γ ER , learning rate α). In Sec. 5, we assumed  a tuning budget that allows only one value for each of λ ER , γ ER , and α. By not tuning these hyperparameters, we greatly reduced our hyperparameter search space. Since ER has little effect on ID task performance, tuning based on ID task performance is unlikely to have helped anyway. ER works better on OOD data, but it also does not make sense to tune based on OOD task performance (otherwise, it would not be OOD). Though the ER hyperparameters chosen via intrinsic evaluation generally improved OOD task performance, we seek to verify their effectiveness compared to other possible hyperparameter values.
In Table 10 (in the appendix), we report sentiment analysis OOD (Amazon, Yelp, Movies) task performance, while varying each of the three hyperparameters. We include a Mean column, which averages the Amazon/Yelp/Movies columns. Our hyperparameters chosen via ER loss curves are highlighted in blue . For λ ER , 1 (ours) and 100 yield very similar Mean results, while considerably beating the other three values. For γ ER , we see the same trend for 100 (ours) and 10. For α, 2e−5 (ours) vastly outperforms other values in all columns. These results validate the utility of ER-TEST's intrinsic evaluation for low-resource ER hyperparameter tuning.

A.4 Details for Functional Tests
In this section, we provide details for different functional tests listen in Section 3.2.3. We breakdown each subcategory of functional tests and show performances of different ER criteria on those individual tests. For functional tests on the sentiment  analysis task, refer to Table 11. NLI functional tests are listed in Table 12.

A.5 Details for Instance Prioritisation Experiments
In this section, we provide further implementation details for confidence-based instance prioritisation experiments as described in Section 4.2. Given that we have 3-seed runs for the None model in Table 1, we extract the confidence scores based on the given metric (LC or HC), and then average these confidence scores across the 3 seed runs to obtain a single score for every instance. This process is done for training set instances only. This is followed by ranking each instance by the aggregated confidence metric and selecting the top k% of samples from this ranking. For experiments with random sampling based prioritisation, we generate 3 random subsets selected in a uniform manner.
While training in this setting, we ensure that within each batch, certain (one third to be specific) set of instances have available rationales. For these instances, we calculate the ER loss L ER , whereas, for the rest of the instances in the batch, we compute the task loss L task . All prioritisation settings are trained with 3 different model seeds and the aggregated results for ID and OOD datasets are shown in Table 3