Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling

Ensembling BERT models often significantly improves accuracy, but at the cost of significantly more computation and memory footprint. In this work, we propose Multi-CLS BERT, a novel ensembling method for CLS-based prediction tasks that is almost as efficient as a single BERT model. Multi-CLS BERT uses multiple CLS tokens with a parameterization and objective that encourages their diversity. Thus instead of fine-tuning each BERT model in an ensemble (and running them all at test time), we need only fine-tune our single Multi-CLS BERT model (and run the one model at test time, ensembling just the multiple final CLS embeddings). To test its effectiveness, we build Multi-CLS BERT on top of a state-of-the-art pretraining method for BERT (Aroca-Ouellette and Rudzicz, 2020). In experiments on GLUE and SuperGLUE we show that our Multi-CLS BERT reliably improves both overall accuracy and confidence estimation. When only 100 training samples are available in GLUE, the Multi-CLS BERT_Base model can even outperform the corresponding BERT_Large model. We analyze the behavior of our Multi-CLS BERT, showing that it has many of the same characteristics and behavior as a typical BERT 5-way ensemble, but with nearly 4-times less computation and memory.


Introduction
BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) is one of the most widely-used language model (LM) architectures for natural language understanding (NLU) tasks. We often fine-tune the pretrained BERT or its variants such as RoBERTa (Liu et al., 2019) so that the LMs learn to aggregate all the contextualized word embeddings into a single CLS embedding for a downstream text classification task. * During fine-tuning, different initializations and different training data orders significantly affect BERT's generalization performance, especially with a small training dataset (Dodge et al., 2020;Zhang et al., 2021a;Mosbach et al., 2021). One simple and popular solution to the issue is to finetune BERT model multiple times using different random seeds and ensemble their predictions to improve its accuracy and confidence estimation. Although very effective, the memory and computational cost of ensembling a large LM is often prohibitive (Xu et al., 2020;Liang et al., 2022). Naturally, we would like to ask, "Is it possible to ensemble BERT models at no extra cost?" To answer the question, we propose Multi-CLS BERT, which enjoys the benefits of ensembling without sacrificing efficiency. Specifically, we input the multiple CLS tokens to BERT and encourage the different CLS embeddings to aggregate the information from different aspects of the input text. As shown in Figure 1, the proposed Multi-CLS BERT shares all the hidden states of the input text and only ensembles different ways of aggregating the hidden states. Since the input text is usually much longer than the number of inputted CLS embeddings, Multi-CLS BERT is almost as efficient as the original BERT.
Allen-Zhu and Li (2020) discovered that the key of an effective ensembling model is the diversity of individual models and the models trained using different random seeds have more diverse predictions compared to simply using dropout (Srivastava et al., 2014;Gal and Ghahramani, 2016) or averaging the weights of the models during training (Fort et al., 2019). To ensure the diversity of CLS embeddings without fine-tuning Multi-CLS BERT using multiple seeds, we propose several novel diversification techniques. For example, we insert different linear layers into the transformer encoder for different CLS tokens. Furthermore, we propose a novel reparametrization trick to prevent the linear layers from learning the same weights during fine-tuning.
We test the effectiveness of these techniques by modifying the multi-task pretraining method proposed by Aroca-Ouellette and Rudzicz (2020), which combines four self-supervised losses. In our experiments, we demonstrate that the resulting Multi-CLS BERT can significantly improve the accuracy on GLUE (Wang et al., 2019b) and Su-perGLUE (Wang et al., 2019a), especially when the training sizes are small. Similar to the BERT ensemble model, we further show that multiple CLS embeddings significantly reduce the expected calibration error, which measures the quality of prediction confidence, on the GLUE benchmark.

Main Contributions
• We propose an efficient ensemble BERT model that does not incur any extra computational cost other than inserting a few CLS tokens and linear layers into the BERT encoder. Furthermore, we develop several diversification techniques for pretraining and fine-tuning the proposed Multi-CLS BERT model.
• We improve the GLUE performance reported in Aroca-Ouellette and Rudzicz (2020) using a better and more stable fine-tuning protocol and verify the effectiveness of its multi-task pretraining methods in GLUE and SuperGLUE with different training sizes.
• Building on the above state-of-the-art pretraining and fine-tuning for BERT, our experiments and analyses show that Multi-CLS BERT signif-icantly outperforms the BERT due to its similarity to a BERT ensemble model. The comprehensive ablation studies confirm the effectiveness of our diversification techniques.

Method
In sections 2.1 and 2.2, we first review its state-ofthe-art pretraining method from Aroca-Ouellette and Rudzicz (2020). In Section 2.3, we modify one of its losses, quick thoughts (QT), to pretrain our multiple embedding representation. In Section 2.4, we encourage the CLS embeddings to capture the fine-grained semantic meaning of the input sequence by adding hard negatives during the pretraining. To diversify the CLS embeddings, we modify the transformer encoder in Section 2.5 and propose a new reparametrization method during the fine-tuning in Section 2.6.

Multi-task Pretraining
After testing many self-supervised losses, Aroca-Ouellette and Rudzicz (2020) find that combining the masked language modeling (MLM) loss, TFIDF loss, sentence ordering (SO) loss (Sun et al., 2020), and quick thoughts (QT) loss (Logeswaran and Lee, 2018) could lead to the best performance. The MLM loss is to predict the masked words and the TFIDF loss is to predict the importance of the words in the document. Each input text sequence consists of multiple sentences. They swap the sentence orders in some input sentences and use the CLS embedding to predict whether the order is swapped in the SO loss. Finally, QT loss is used to encourage the CLS embeddings of the consecutive sequences to be similar.
To improve the state-of-the-art pretraining method, we modify the multi-task pretraining method by using multiple CLS embeddings to represent the input sequence and using non-immediate consecutive sentences as the hard negative. Our training method is illustrated in Figure 2.

Quick Thoughts Loss
Two similar sentences tend to have the same label in a downstream application, so pretraining should aim to pull the CLS embeddings of these similar sentences closer. The QT loss achieves this goal by assuming consecutive text sequences are similar and encouraging their CLS embeddings to be similar. [CLS0] c1 1-2 h1 1-2 hn+3 1-2 c2 1-2 c3 1-2 c4 1-2 c5 1-2 h2 1-2 hn+1 1-2 hn+2 1-2 hn+m+2 1-2 Figure 2: Our MCQT, SO, MLM, and TFIDF loss, which are a modification of multi-task pretraining proposed in Aroca-Ouellette and Rudzicz (2020). The multi-CLS quick thought (MCQT) loss maximizes the CLS similarities between a sequence (sentences 1 and 2) and the next sequence (sentences 3 and 4) while minimizing the CLS similarities to other random sequences and the sequence after the next one (sentences 5 and 6). Notice that sentence 4 is inputted before sentence 3 because the sentence order is swapped for the SO loss.
Aroca-Ouellette and Rudzicz (2020) propose an efficient way of computing QT loss in a batch by evenly splitting each batch with size B into two parts. The first part contains B/2 text sequences randomly sampled from the pretrained corpus, and the second part contains each of the B/2 sentences that are immediately subsequent to those in the first part. Then, for each sequence in the first part, they use the consecutive sequence in the second part as the positive example and the other B/2 − 1 sequences as the negative examples. We can write the QT loss for the sequences containing sentences 1, 2, 3, and 4 as where s is the sentences in the second part of the batch, Logit QT s 1−2 ,s 3−4 = ( c 1−2 ||c 1−2 || ) T c 3−4 ||c 3−4 || is the score for classifying sequence s 3−4 as the positive example, c 1−2 ||c 1−2 || is the L2-normalized CLS embedding for sentences 1 and 2. The normalization is intended to stabilize the pretraining by limiting the gradients' magnitudes.

Multiple CLS Embeddings
A text sequence could have multiple facets; two sequences could be similar in some facets but dissimilar in others, especially when the text sequences are long. The QT loss squeezes all facets of a sequence into a single embedding and encourages all facets of two consecutive sequences to be similar, potentially causing information loss.
Some facets might better align with the goal of a downstream application. For example, the facets that contain more sentiment information would be more useful for sentiment analysis. To preserve the diverse facet information during pretraining, we propose multi-CLS quick thoughts loss (MCQT). The loss integrates two ways of computing the similarity of two sequences. The first way computes the cosine similarity between the most similar facets, and the second computes the cosine similarity between the summations of all facets. We linearly combine the two methods as the logit of the two input sequences: where λ is a constant hyperparameters; c 1−2 k and c 3−4 k are the CLS embeddings of sentences 1-2 and sentences 3-4, respectively.
The first term only considers the most similar facets to allow some facets to be dissimilar. Furthermore, the term implicitly diversifies CLS embed-dings by considering each CLS embedding independently. In contrast, the second term encourages the CLS embeddings to work collaboratively, as in a typical ensemble model, and also let every CLS embedding receive gradients more evenly. Notice that we sum the CLS embeddings before the normalization so that the encoder could predict the magnitude of each CLS embedding as its weight in the summation.
To show that the proposed method can improve the state-of-the-art pretraining methods, we keep the MLM loss and TFIDF loss unchanged. For the sentence ordering (SO) loss, we project the K hidden states h c k into the embedding h SO with the hidden state size D for predicting the sentence order:

Hard Negative
For a large transformer-based LM, distinguishing the next sequence from random sequences could be easy. The LM can achieve low QT loss by outputting nearly identical CLS embeddings for the sentences with the same topic while ignoring the fine-grained semantic information (Papyan et al., 2020). In this case, using multiple CLS embeddings might become underutilized.
The hard negative is a common method of adjusting the difficulties of the contrastive learning (Baldini Soares et al., 2019;Cohan et al., 2020). Our way of collecting hard examples is illustrated in the bottom-left block of Figure 2. To efficiently add the hard negatives in the pretraining, we split the batch into three parts. For each sequence in the first part, we would use its immediate next sequence in the second part as the positive example, use the sequence after the next one in the third part as the hard negative, and use all the other sequences in the second or the third part as the easy negatives. We select such sequence after the next one as our hard negatives because the sequence usually share the same topic with the input sequence but is more likely to have different fine-grained semantic facets compared to the immediate next sequence.
After adding the hard negative, the modified QT loss of the three consecutive sequences becomes where MCQT refer to multi-CLS quick thoughts, {s 3−4 , ..., s 5−6 , ...} are all the sequences in the second and the third part, and {s 3−4 , ..., s 1−2 , ...} are all the sequences in the first and the second part.

Architecture-based Diversification
Initially, we simply input multiple special CLS tokens ([C1], ..., [CK]) after the original CLS token, [CLS 0 ], and take the corresponding hidden states as the CLS embeddings, but we found that the CLS embeddings quickly become almost identical during the pretraining. Subsequently, instead of using the same final transformation head H QT for all CLS hidden states, we use a different linear layer L O,k in the final head H M C k to transform the hidden state h c k for the kth CLS. We set the bias term in L O,k to be the constant 0 because we want the differences between the CLS to be dynamic and context-dependent.
Nevertheless, even though the resulting CLS embeddings c k = H M C k (h c k ) are differentiated, the hidden states h c k before the transformation head usually still collapse into almost identical embeddings.
To solve the collapsing problem, we insert multiple linear layers L l,k into the transformer encoder. In Figure 3, we illustrate our encoder architecture built on the BERT Base model. After the 4th transformer layer, we insert the layers L 4,k to transform the hidden states before inputting them to the 5th layer. Similarly, we insert L 8,k between the 8th transformer layer and 9th transformer layer. For BERT Large , we insert L l,k (.) after layer 8 and layer 16. Notice that although the architecture looks similar to the adapter (Houlsby et al., 2019) or prefix-tuning (Li andLiang, 2021), our purpose is to diversify the CLS embeddings rather than freezing parameters to save computational time.

Fine-Tuning
To avoid overfitting and increasing computational overhead, we pool multiple CLS hidden states into the single CLS embedding for downstream task fine-tuning. As a result, we can use the same classifier architecture on top of Multi-CLS BERT and BERT, which also simplifies their comparison.
We discover that simply summing all the CLS hidden states still usually makes the hidden states and the inserted linear layers (e.g., L O,k ) almost identical after fine-tuning. To avoid collapsing, we aggregate the CLS hidden states by proposing a novel re-parameterization trick: where However, gradient descent would not allow the model to constantly output the zero vector, so L F T O,k remains different during the fine-tuning.

Experiments
The parameters of neural networks are more restricted as more training samples are available (MacKay, 1995) and the improvement of deep ensemble models comes from the diversity of individual models (Fort et al., 2019), so the benefits of ensembling are usually more obvious when the training set size is smaller. Therefore, in addition to using the full training dataset, we also test the settings where the models are trained by 1k samples (Zhang et al., 2021a) or 100 samples from each task in GLUE (Wang et al., 2019b) or Super-GLUE (Wang et al., 2019a). Another benefit of the 1k-and 100-sampling settings is that the average scores would be significantly influenced by most datasets rather than by only a subset of relatively small datasets (Card et al., 2020).

Experiment Setup
To accelerate the pretraining experiments, we initialize the weights using the pretrained BERT models (Devlin et al., 2019) and continue the pretraining using different loss functions on Wikipedia 2021 and BookCorpus (Zhu et al., 2015).
All of the methods are based on uncased BERT as in Aroca-Ouellette and Rudzicz (2020). We compare the following methods: • MTL: Pretraining using the four losses selected in Aroca-Ouellette and Rudzicz (2020): MLM, QT, SO, and TFIDF. We remove the continue learning procedure used in ERNIE (Sun et al., 2020) because we find that simply summing all the losses leads to better performance (see our ablation study in Section 3.3).
• Ours (K=1): We set K = 1 in our method to verify the effectiveness of using multiple embeddings. During fine-tuning, the CLS embedding is a linear transformation of the single facet CLS = L O,1 (h f 1 ). The GLUE and SuperGLUE scores are significantly influenced by the pretraining random seeds (Sellam et al., 2021) and fine-tuning random seeds (Dodge et al., 2020;Zhang et al., 2021a;Mosbach et al., 2021). To stably evaluate the performance of different pretraining methods, we pretrain models using four random seeds and fine-tune each pretrained model using four random seeds, and report the average performance on the development set across all 16 random seeds. To further stabilize the fine-tuning process and reach better performance, we follow the fine-tuning suggestions from Zhang et al. (2021a) and Mosbach et al. (2021), including training longer, limiting the gradient norm, and using Adam (Kingma and Ba, 2015) with bias term and warmup.

Main Results
Our results are presented in We observe that λ = 0.1 usually performs well, which justifies the inclusion of both the highest logit and average logit in Equation 2. The λ = 0 model has significantly worse performance only in BERT Large model. This suggests that the benefits of Multi-CLS BERT depend on our pretraining method and maximizing the highest logit stabilizes the pretraining of a larger model.

Ablation Study
In our ablation studies, we would like to test the effectiveness of the design choices in our baseline MTL and our best model, Ours (K=5, λ = 0.1).  ent seeds and averaging their prediction probabilities for each class in each task.
• Ensemble on FT Seeds: Fine-tuning Ours (K=1) or Ours (K=5, λ = 0.1) using 5 different seeds and averaging their prediction probabilities. Our results are presented in Table 2. We can see that continuing training using MLM only loss degrades the performance, which indicates that our improvement does not come from training BERT longer. Removing QT loss results in mixed results. The better performance of MTL compared to CMTL+ suggests that the continual training technique used in Aroca-Ouellette and Rudzicz (2020) is harmful with our evaluation settings.
Removing the inserted layers (No Inserted Layers) or removing the re-parametrization trick (Sum Aggregation) makes the performance of Ours (K=5, λ = 0.1) close to the Ours (K=1) baseline. This result highlight the importance of diversity of CLS embeddings. The performance of Ours (K=3) and Ours (K=10) is usually better than Ours (K=1), but are worse than Ours (K=5). In both BERT Base and BERT Large models, removing hard negatives degrades the GLUE scores but slightly increases the SuperGLUE scores.
In GLUE 100 and 1k, we do not get good results by using other efficient ensembling methods   such as SWA and Ensemble on Dropouts. This suggests that the gradient descent trajectory and different dropout maps might not produce prediction diversity sufficient for an effective BERT ensemble model (Fort et al., 2019). On the other hand, ensembling the models that are fine-tuned using different random seeds indeed boosts the performance at the expense of high computational costs. The ensembled Multi-CLS BERT (Ensemble on FT Seeds K=5) still outperforms the ensembled K=1 baseline, but ensembling makes their performance differences smaller. These results imply that the improvements of Multi-CLS BERT overlap with the improvements of a BERT ensemble model.

Ensembling Analysis
We compare the inference time and expected calibration error (ECE) (Naeini et al., 2015) of using multiple CLS embeddings, using a single CLS embedding, and ensembling BERT models with different fine-tuning seeds in Table 3. A lower ECE means a better class probability estimation. For example, if a model outputs class 1 with 0.9 probability for 100 samples, ECE = 0 means that 90 samples among them are indeed class 1. Table 3 shows that Ours (K=5) is much faster than the BERT ensemble and almost as efficient as Ours (K=1), because a BERT ensemble needs to run for multiple forward passes and we reduce the maximal sentence length by 5 in Ours (K=5). Additionally, the ECE of Ours (K=5) is lower than Ours (K=1) but not as low as the ECE from ensembling BERT models with different fine-tuning seeds. That is, without significantly increasing inference time, ensembling multiple CLS embeddings improves the output confidence, even though not as much as ensembling BERT models.
Next, we analyze the correlation of uncertainty estimation from different methods in Table 4. When ensembling BERT models with different dropout maps (Dropout) or different fine-tuning seeds (ENS), we can estimate the prediction uncertainty by the variance of the prediction probability from each individual BERT model. We can also use one minus prediction probability as the uncertainty (Least). In Multi-CLS, we measure the disagreement among the CLS embeddings as the uncertainty 2 and would like to see how many top-20% most uncertain samples from the disagreement of CLS embeddings are also the top-20% most uncertain samples for a BERT ensemble model. Table 4 reports the ratio of the number of the overlapping uncertain samples from two estimation methods to the number of 20% samples in the development set. We can see that the ratio from Multi-CLS BERT and the BERT ensemble model (Multi-CLS vs ENS) is close to the ratios from other uncertainty estimations and the BERT ensemble model (Dropout vs ENS, Least vs ENS, and ENS vs ENS). This shows that different CLS embeddings can classify the uncertain samples differently, as is the case for the different BERT models in a BERT ensemble model. In Appendix C, we visualize the CLS embeddings of some uncertain samples to show how different CLS embeddings solve a task in different ways.

Related Work
Due to its effectiveness, ensembling BERT in a better or more efficient way has recently attracted researchers' attention. Nevertheless, the existing approaches often need to rely on distillation (Xu et al., 2020;Matsubara et al., 2022;Zuo et al., 2022) or still require significant extra computational cost during training and testing (Kobayashi et al., 2022;Liang et al., 2022).
Some recent vision models can also achieve ensembling almost without extra computational cost by sharing the weights (Wen et al., 2020), partition-2 See Appendix B.3 for details ing the model into subnetworks (Havasi et al., 2021;Zhang et al., 2021b), or partitioning the embeddings (Lavoie et al., 2022). However, it is unknown if the approaches are applicable to the pretraining and fine-tuning of language models. Similar to Multi-CLS BERT, mixture of softmax (MoS) (Yang et al., 2018) also uses multiple embeddings to improve the pretraining loss. Recently, Narang et al. (2021); Anonymous (2021) have found that MoS is one of the few modifications that can improve on the original BERT architecture on the NLU benchmarks. Nevertheless, Narang et al. (2021) also point out that MoS requires significant extra training cost to compute the multiplication between each hidden state and all the word embeddings in the vocabulary. Chang et al. (2021) propose to use multiple embeddings to represent the sentence and demonstrate its improvement over the single embedding baseline on unsupervised sentence similarity tasks. Furthermore, their non-negative sparse coding loss also encourages multiple sentence embeddings to collaborate during pretraining, which is similar to our Equation 2. Nevertheless, our loss is more computationally efficient and is designed to improve downstream supervised tasks rather than similarity tasks.
Another approach represents a document using sentence embeddings (Iter et al., 2020;Mysore et al., 2022) or contextualized word embeddings (Khattab and Zaharia, 2020; Luan et al., 2021) for information retrieval applications. However, the goal of this approach is to improve the representation of a relatively long document and it is unknown if its benefits could be extended to the GLUE tasks that require fine-tuning and often involve only one or two sentences.

Conclusion
In this work, we propose representing the input text using K CLS embeddings rather than using the single CLS embedding in BERT. Compared to BERT, Multi-CLS BERT significantly increases the GLUE and SuperGLUE scores and reduces the expected calibration error in GLUE, while its only added cost is to reduce the maximal text length by K and add a little extra time for computing the inserted linear transformations. Therefore, we recommend the wide use of multiple CLS embeddings for the almost free performance gain.
To solve the collapsing problem of CLS embed-dings, we modify the pretraining loss, BERT architecture, and fine-tuning loss. The ablation study shows that all of these modifications contribute to the performance improvement of Multi-CLS BERT. In our analyses for investigating the source of the improvement, we find that a) ensembling the original BERT leads to greater improvement than ensembling the Multi-CLS BERT and b) the disagreement of different CLS embeddings highly correlates with the disagreement of the BERT models from different fine-tuning seeds. Both findings support our perspective that Multi-CLS BERT is an efficient ensembling method.

Acknowledgement
We thank Jay Yoon Lee and the anonymous reviewers for their constructive feedback.

Limitations
Our methods are evaluated using BERT as many previous recent work such as Aroca-Ouellette and Rudzicz (2020)  Our conclusion mainly draws from the overall scores of GLUE or SuperGLUE benchmarks, which only include English datasets and might contain some dataset selection bias (Dehghani et al., 2021).
Although much more efficient, Multi-CLS BERT is still worse than the classic BERT ensemble model in terms of expected calibration error and accuracy when more training data are available (e.g., in GLUE 1k). We also do not know if Multi-CLS BERT could provide efficient and high-quality uncertainty estimation for other applications such as active learning (Pop and Fulop, 2018).

Ethical and Broader Impact
Multi-CLS BERT can provide better confidence estimation compared to BERT while better efficiency compared to the classic BERT ensemble. This work might inspire prospective efficient ensembling approaches that produce more robust predictions (Clark et al., 2019) with lower the energy consumption.
On the other hand, the readers of the paper might not notice the limitations of the study (e.g., the confidence estimation of Multi-CLS BERT is still sometimes far behind the classic BERT ensemble model) and mistakenly believe that Multi-CLS BERT has all the benefits of the classic BERT ensemble model.

A Appendix Overview
In the appendix, we first describe the details of our methods and evaluation protocol in Appendix B. Then, we visualize the disagreement of CLS embeddings of some samples in Appendix C. Finally, we compare the performance of individual tasks in Appendix D.

B Experiment Details
We first describe the architecture details and pretraining details of our methods and baselines in Appendix B.1. Then, we list the hyperparameter setup in the fine-tuning in Appendix B.2. Finally, we explain the details of the ensemble baselines and their related analyses in Appendix B.3. We will release our code to further increase the reproducibility after the paper is accepted.

B.1 Our Models and Baselines
The models built on BERT Base are pretrained using two billion tokens and each batch contains 30 sequences. The models built on BERT Large are pretrained using one billion tokens and each batch contains 48 sequences. The learning rate is 2 · 10 −5 and the warmup ratio is 0.001 for the pretraining stage.
We implement Multi-CLS BERT by modifying the code of Aroca-Ouellette and Rudzicz (2020) 3 . We use [unused0] -[unused(K-1)] tokens in the original BERT tokenizer as our input CLS tokens [C1] - [CK]. We still keep the original CLS tokens to increase the comparability with the MTL baseline.
We use NVIDIA GeForce RTX 2080, 1080, and TITAN X, M40 GPUs for the BERT Base experiments and use GeForce RTX 8000 and Tesla M40 for the BERT Large experiments. In Table 1, the model size excludes the top classifier parameters used in each task.
We test CMTL+ using the default hyperparameters of Aroca-Ouellette and Rudzicz (2020) and we do not try different hyperparameters or different schedules of pretraining losses. No Inserted Layers only removes the L l,k (.) while still using different H M C k on top during pretraining. SWA averages the weights of every model checkpoint that is evaluated using the validation dataset.

B.2 Fine-tuning
We start from the default evaluation hyperparameters used in Aroca-Ouellette and Rudzicz (2020) and modify the settings based on the suggestions from Zhang et al. (2021a) and Mosbach et al. (2021). We find that the best hyperparameters depend on the training size. For example, batch size 16 works well in GLUE Full but is much worse than batch size 4 in GLUE 100. Furthermore, the performance of the default hyperparameters on some tasks is suboptimal or unstable even after averaging the performance from 16 trials. Therefore, we coarsely tune the hyperparameters to maximize and stabilize the performance of the Ours (K=1) baseline under the memory and computational time constraints in our GPUs. The preliminary results suggest that the hyperparameters also maximize the performance of MTL.
Next, we list fine-tuning hyperparameters for all the tasks 4 . Our fine-tuning stops after 20 epochs, 60k batches, or consecutive 10k batches without a validation improvement (whichever comes first). We use the first 5k validation samples to select the best fine-tuned model checkpoints for the evaluation. The maximal gradient norm is 1. The maximal length for sentences and CLS tokens is 128 for GLUE and 256 for SuperGLUE.
For each task, we select the best learning rate from c · 10 −5 and c = 1, 2, 3, 4, 5, 7. When running large datasets in GLUE Full and SuperGLUE Full (MNLI, QQP, QNLI, SST-2, BoolQ, MultiRC, and WiC) using BERT Large , we use learning rates c = 2, 4, 6, 8, 10, 14 to accelerate the training. The batch sizes for GLUE 100, 1k, Full are 4, 8, 16, respectively. The batch size for SuperGLUE is 4 except that the BERT Large models use 8 in Super-GLUE 1k and Full. For BERT Base , the warmup ratio is 0.1. For BERT Large , the warmup ratio is 0.2 and the weight decay is 10 −6 .
For each fine-tuning random seed, we randomly select a different subset in the settings where only 100 or 1k training samples are available. For the datasets with less than 500 training samples in Su-perGLUE and SuperGLUE 1k (i.e., CB and COPA), we repeat the experiments 32 times to further stabilize the scores. For the pre-trained BERT baseline, we use 16 fine-tuning random seeds. To reduce the computational cost, we use two pretraining random seeds and four fine-tuning random seeds in our ablation study in Table 2. Compared to other tasks, ReCoRD needs to be trained much longer than other tasks in Super-GLUE, so we only use one fine-tuning seed for each of the four pretrained models with different seeds. Our fine-tuning stops after 600k batches (BERT Base ) / 300k batches (BERT Large ) or consecutive 160k batches without a validation improvement (whichever comes first).
To stabilize the performance of each model on ReCoRD, we use the first 40k validation samples to select the best fine-tuned model checkpoints. We set batch size as 8 and learning rate as 1 · 10 −5 for BERT Base . For BERT Large , we set batch size as 32 and learning rate as 2 · 10 −5 .

B.3 Ensemble Models
Ensemble on FT Seeds (K=1) in Table 2 is the same as Ensemble of Ours (K=1) in Table 3. Ensemble on FT Seeds (K=5) in Table 2 is the same as ENS in Table 4. Ensemble on Dropouts in Table 2 is the same as Dropout in Table 4. All results are the average of four models that use four different pretrained models and the best learning rate among c · 10 −5 (c = 1, 2, 3, 4, 5, 7) in the finetuning stage.
In Table 3, we compute the expected calibration error (ECE) (Naeini et al., 2015) by where acc(j) is the model accuracy in the jth bin B j , N is the number of validation samples, and conf(j) = 1 |B j | x∈B j max y P (y|x) is the average of the highest prediction probability P (y|x) in the jth bin. We put the samples into 10 equal-size bins according to their highest prediction probability max y P (y|x).
In Table 3, we use Tesla M40 to measure the inference time of the models built on BERT Base . We set batch size 16 and run 1000 batches to get the average inference time of one batch in every GLUE task. We repeat the experiments five times and report their average and standard error. For the ensemble model, we assume the time of averaging multiple prediction probabilities is negligible and directly multiply the inference time of Ours (K=1) by 5.
In Table 4, we would like to see if CLS embeddings disagree with each other as other ensemble baselines did. In Multi-CLS, we compute the uncertainty of each sample x as the average variance of prediction probability of each CLS embedding mean l (var k P (y = l|x, k)) and estimate the prediction probability of the kth CLS embedding by is the CLS embedding of the input x after fine-tuning, and q i,k = is the ith class embedding for the kth CLS embedding, which is computed by averaging the kth CLS embeddings of the input x with the ith class label.
In Table 4, the two ensemble models for ENS vs ENS use the same set of 5 fine-tuning seeds and the two Ours (K=5, λ = 0.1) pretrained with different random seeds. Both uncertainty estimation models for Multi-CLS vs ENS, Dropout vs ENS, and Least vs ENS are based on the same pretrained Ours (K=5, λ = 0.1) model. Table 5-16 compare the CLS embeddings of Ours (K=1) and Ours (K=5, λ = 0.1) after fine-tuning to illustrate how different CLS embeddings capture distinct aspects of an input sentence in solving a task. For each task, we select one sample (a sentence or a sentence pair) from the validation set whose CLS embeddings disagree with each other.

C Visualization of CLS embeddings
For each selected sample, we visualize its nearest-neighboring sentences in the validation set with respect to each CLS embedding. The nearest neighbors for the kth CLS embedding are determined by the cosine similarity between the respective kth CLS embedding of the input sentence and other sentences. Beside each sentence or sentence pair, we show their ground truth label and the model's prediction.
In Ours (K=5, λ = 0.1), two representative sentences are selected from the top-three nearest neighbors for each CLS, and each CLS is manually annotated with terms that summarize those aspects that are shared by the neighbors and relate to the query sentence. For comparison, accompanying tables show the top-ten nearest neighbors for Ours (K=1).
In almost all the classification tasks, we observe that CLS 3 and CLS 5 vote for the same class (i.e., their embeddings are close to the neighbors with the same class prediction). On the other hand, CLS 1 and CLS 4 vote for another class in these examples where the CLS embeddings disagree. The observation suggests that the similarity of CLS embeddings after the pretraining stage correlates with their similarity after the fine-tuning.    Table 7: Visualization of Ours (K=5, λ = 0.1) using a sample in SST-2. The neighbors from CLS 2 and 4 share the same "postive, but negative" template as in the query. Like the query, CLS 1, 3, and 5 capture the positive aspects. Some CLSs also capture the semantic aspects of the query such as script, sci-fi, or liquid.  equivalent equivalent S1: A man arrested for allegedly threatening to shoot and kill a city councilman from Queens was ordered held on $100,000 bail during an early morning court appearance Saturday. S2: The Queens man arrested for allegedly threatening to shoot City Councilman Hiram Monserrate was held on $100,000 bail Saturday, a spokesman for the Queens district attorney said. CLS-space Neighbors (K=5): CLS 1 equivalent equivalent S1: Myanmar's pro-democracy leader Aung San Suu Kyi will return home late Friday but will remain in detention after recovering from surgery at a Yangon hospital, her personal physician said. S2: Myanmar's pro-democracy leader Aung San Suu Kyi will be kept under house arrest following her release from a hospital where she underwent surgery, her personal physician said Friday.

Politics
Justice not equivalent not equivalent S1: Bob Richter, a spokesman for House Speaker Tom Craddick, had no comment about the ruling. S2: Bob Richter, spokesman for Craddick, R-Midland, said the speaker had not seen the ruling and could not comment. CLS 2 equivalent equivalent S1: They were being held Sunday in the Camden County Jail on $100,000 bail. S2: They remained in Camden County Jail on Sunday on $100,000 bail. Thousands Crime or threat equivalent equivalent S1: "More than 70,000 men and women from bases in Southern California were deployed in Iraq. S2: In all, more than 70,000 troops based in Southern California were deployed to Iraq. CLS 3 equivalent equivalent S1: Robert Walsh, 40, remained in critical but stable condition Friday at Staten Island University Hospital's north campus. S2: Walsh, also 40, was in critical but stable condition at Staten Island University Hospital last night. Time equivalent equivalent S1: Blair's Foreign Secretary Jack Straw was to take his place on Monday to give a statement to parliament on the European Union. S2: Blair's office said his Foreign Secretary Jack Straw would take his place on Monday to give a statement to parliament on the EU meeting the prime minister attended last week. CLS 4 not equivalent not equivalent S1: Franklin County Judge-Executive Teresa Barton said a firefighter was struck by lightning and was taken to the Frankfort Regional Medical Center. S2: A county firefighter, was struck by lightning and was in stable condition at Frankfort Regional Medical Center. Comments Medical or justice equivalent equivalent S1: Myanmar's pro-democracy leader Aung San Suu Kyi will return home late Friday but will remain in detention after recovering from surgery at a Yangon hospital, her personal physician said. S2: Myanmar's pro-democracy leader Aung San Suu Kyi will be kept under house arrest following her release from a hospital where she underwent surgery, her personal physician said Friday. CLS 5 equivalent equivalent S1: Unable to find a home for him, a judge told mental health authorities they needed to find supervised housing and treatment for DeVries somewhere in California. S2: The judge had told the state Department of Mental Health to find supervised housing and treatment for DeVries somewhere in California.
Court's ruling equivalent equivalent S1: A former employee of a local power company pleaded guilty Wednesday to setting off a bomb that knocked out a power substation during the Winter Olympics last year. S2: A former Utah Power meter reader pleaded guilty Wednesday to bombing a power substation during the 2002 Winter Olympics. Table 9: Visualization of Ours (K=5, λ = 0.1) using a sample in MRPC. The neighbors from CLS 2, 3, and 5 focus on different aspects of the query. The neighbors from CLS 1 and 4 are someone's comments as in the query and might not be equivalent. Several CLSs are also related to justice.
Task: MRPC Prediction Label Sentence Pair Query: equivalent equivalent S1: A man arrested for allegedly threatening to shoot and kill a city councilman from Queens was ordered held on $100,000 bail during an early morning court appearance Saturday. S2: The Queens man arrested for allegedly threatening to shoot City Councilman Hiram Monserrate was held on $100,000 bail Saturday, a spokesman for the Queens district attorney said. CLS-space Neighbors (K=1): equivalent equivalent S1: The Justice Department Aug. 19 gave pre-clearance for the Oct. 7 date for the election to recall Gov. Gray Davis, saying it would not affect minority voting rights. S2: The Justice Department on Aug. 19 sanctioned the Oct. 7 date for recall election, saying it would not affect voting rights. equivalent equivalent S1: The worm attacks Windows computers via a hole in the operating system, an issue Microsoft on July 16 had warned about. S2: The worm attacks Windows computers via a hole in the operating system, which Microsoft warned of 16 July. equivalent equivalent S1: O'Brien was charged with leaving the scene of a fatal accident, a felony. S2: Bishop Thomas O'Brien, 67, was booked on a charge of leaving the scene of a fatal accident. equivalent equivalent S1: "There is no conscious policy of the United States, I can assure you of this, to move the dollar at all," he said. S2: He also said there is no conscious policy by the United States to move the value of the dollar. equivalent equivalent S1: The AFL-CIO is waiting until October to decide if it will endorse a candidate. S2: The AFL-CIO announced Wednesday that it will decide in October whether to endorse a candidate before the primaries. equivalent equivalent S1: Speaking for the first time yesterday, Brigitte's maternal aunt said his family was unaware he had was in prison or that he had remarried. S2: Brigitte's maternal aunt said his family was unaware he had been sent to prison, or that he had remarried in Sydney.
equivalent not equivalent S1: Rosenthal is hereby sentenced to custody of the Federal Bureau of prisons for one day with credit for time served," Breyer said to tumultuous cheers in the courtroom. S2: "Rosenthal is hereby sentenced to custody of the Federal Bureau of Prisons for one day with credit for time served." equivalent equivalent S1: Police say CIBA was involved in the importation of qat, a narcotic substance legal in Britain but banned in the United States. S2: Mr McKinlay said that CIBA was involved in the importation of qat, a narcotic substance legal in Britain but banned in the US. equivalent equivalent S1: Judge Craig Doran said it wasn't his role to determine if Hovan was "an evil man" but maintained that "he has committed an evil act." S2: Judge Craig Doran said he couldn't determine if Hovan was "an evil man" but said he "has committed an evil act." equivalent equivalent S1: But MTA officials appropriated the money to the 2003 and 2004 budgets without notifying riders or even the MTA board members considering the 50-cent hike, Hevesi found. S2: MTA officials appropriated the surplus money to later years' budgets without notifying riders or the MTA board members when the 50-cent hike was being considered, he said. contradiction contradiction S1: There is very little left of old Ocho the scant remains of Ocho Rios Fort are probably the oldest and now lie in an industrial area, almost forgotten as the tide of progress has swept over the town. S2: There is nothing left of the Ocho Rios Fort. CLS-space Neighbors (K=5): CLS 1 neutral contradiction S1: After the purge of foreigners, only a few stayed on, strictly confined to Dejima Island in Nagasaki Bay. S2: A few foreigners were left free after the purge on foreigners. Size or quantity neutral neutral S1: 'Publicity.' Lincoln removed his great hat, making a small show of dusting it off. S2: Lincoln took his hat off. CLS 2 neutral neutral S1: There is no tradition of clothes criticism that includes serious analysis, or even of costume criticism among theater, ballet, and opera critics, who do have an august writerly heritage. S2: Clothes criticism is not serious.
Historical places or heritage Negation neutral neutral S1: All of the islands are now officially and proudly part of France, not colonies as they were for some three centuries. S2: The islands are part of France now instead of just colonies. CLS 3 contradiction neutral S1 : And yet, we still lack a set of global accounting and reporting standards that reflects the globalization of economies, enterprises, and markets. S2: The globalization of economies is not reflected in global accounting standards.

Industry
Region Negation contradiction contradiction S1: The technology used to capture and evaluate information in response to the RFP permits LSC to compile and assess key information about the delivery system at the program, state, regional, and national level. S2: There is no way for the LSC to compile information about delivery systems. CLS 4 neutral neutral S1: Scotland became little more than an English county. S2: Scotland was hardly better than an English county.

Historical places
Minimization or negation neutral neutral S1: Just as in ancient times, without the River Nile, Egypt could not exist. S2: Without the Nile River, Egypt could not exist.
CLS 5 contradiction neutral S1: Beside the fortress lies an 18th-century caravanserai, or inn, which has been converted into a hotel, and now hosts regular folklore evenings of Turkish dance and music. S2: The 18th century caravanserai is now a hotel.

Buildings or properties
Contrast contradiction contradiction S1: Diamonds are graded from D to X, with only D, E, and F considered good, D being colorless or river white, J slightly tinted, Q light yellow, and S to X yellow. S2: There is no difference between diamonds, all having the same properties. contradiction contradiction S1: There is very little left of old Ocho the scant remains of Ocho Rios Fort are probably the oldest and now lie in an industrial area, almost forgotten as the tide of progress has swept over the town. S2: There is nothing left of the Ocho Rios Fort. CLS-space Neighbors (K=1): contradiction contradiction S1: It was utterly mad. S2: It was perfectly normal. contradiction neutral S1: Fixing current levels of damage would be impossible. S2: Fixing the damage could never be done. contradiction contradiction S1: It was still night. S2: The sun was blazing in the sky, darkness nowhere to be seen. contradiction contradiction S1: That's their signal S2: That isn't their signal. contradiction contradiction S1: It is extremely dangerous to Every trip to the store becomes a temptation. S2: Even with every trip to the store, it never becomes a temptation.

contradiction
contradiction S1: The Revolutionaries couldn't be dissuaded from destroying most of the cathedral's statues, although 67 were saved (many of the originals are now housed in the Musée de l'Oeuvre Notre-Dame next door). S2: All of the cathedrals statues were saved by the Revolutionaries. contradiction contradiction S1: It was deserved. S2: It was not deserved at all contradiction entailment S1: And far, far away-lying still on the tracks-was the back of the train. S2: The train wasn't moving but then it started up.
contradiction contradiction S1: Even if you're the kind of traveler who likes to improvise and be adventurous, don't turn your nose up at the tourist offices. S2: There's nothing worth seeing in the tourist offices. contradiction contradiction S1: Cybernetics had always been Derry's passion. S2: Derry knew nothing of cybernetics. not entailment not entailment S1: During which period did Jacksonville become a popular destination for the rich? S2: This highlighted the visibility of the state as a worthy place for tourism. Jacksonville Duration of time not entailment not entailment S1: What brought the downfall of Jacksonville filmmaking? S2: Over the course of the decade, more than 30 silent film studios were established, earning Jacksonville the title of "winter film capital of the world". CLS 4 entailment entailment S1: How did the new king react to the Huguenots? S2: Louis XIV gained the throne in 1643 and acted increasingly aggressively to force the Huguenots to convert. Change entailment not entailment S1: What did Luther begin to experience in 1536? S2: In December 1544, he began to feel the effects of angina. CLS 5 not entailment not entailment S1: What brought the downfall of Jacksonville filmmaking? S2: Over the course of the decade, more than 30 silent film studios were established, earning Jacksonville the title of "winter film capital of the world".
Negative event not entailment not entailment S1: What cycle AC current system did Tesla propose? S2: He found the time there frustrating because of conflicts between him and the other Westinghouse engineers over how best to implement AC power.     2.230 2.200 S1: Man sitting on a bench drink from a mug surrounded by rugs. S2: A man is sitting on one of two red benches and staring into a kiosk.
2.426 2.000 S1: If you can get over the "ick factor," you have an easily-applied source of organic nitrogen fertilizer close at hand. S2: The NPK numbers on the fertilizer represents the percent, by weight, of Nitrogen, P2O5 and K2O, respectively.
2.248 3.400 S1: Try switching to rats; weanling rats if you need something smaller. S2: As mentioned in previous answers, rats and gerbils can be offered instead of mice or in a rotation with mice.

D Performance of Individual Tasks
We report the individual task results of GLUE 100 and 1k in Table 17, the results of GLUE Full in Table 18, the results of SuperGLUE 100 and 1k in Table 19, the results of SuperGLUE Full in Table 20, the results of the top 20% uncertain sample overlapping ratio in Table 22, and the results of ECE in Table 21. In Table 18, we also compare the GLUE score of our MTL baseline with the scores reported in Aroca-Ouellette and Rudzicz (2020). In GLUE 100 and SuperGLUE 100, multiple embeddings are almost always better. In GLUE 1k and Full, the improvement is smaller, so the baselines perform better in some individual tasks. We also observe that different downstream tasks might prefer different lambda.
In Table 21, we compute the p value using Chernoff bound: where µ = (0.2) 2 4N , N is the number of samples in the validation set, δ = 4 i=1 S i µ − 1, and S i is the observed size of overlapping at the ith trial.