Computer Science
Computer Science-2021.gz:143462
Computer Science
d8a96dcf95379855847544784f1daba46b541591
Sentence Embeddings by Ensemble Distillation

This paper contributes a new State Of The Art (SOTA) for Semantic Textual Similarity (STS). We compare and combine a number of recently proposed sentence embedding methods for STS, and propose a novel and simple ensemble knowledge distillation scheme that improves on previous approaches. Our experiments demonstrate that a model trained to learn the average embedding space from multiple ensemble students outperforms all the other individual models with high robustness. Utilizing our distillation method in combination with previous methods, we significantly improve on the SOTA unsupervised STS, and by proper hyperparameter tuning of previous methods we improve the supervised SOTA scores.


A Note on Terminology
Common practice in work on STS is to use the term "unsupervised" to refer to not explicitly utilizing labeled STS data. Approaches using other sorts of labeled data, such as the NLI data for S-BERT, is therefore described as unsupervised in this context. This is somewhat confusing, and not in line with our interpretation of the term unsupervised. Hence, throughout this paper we will use the term "zero-shot" when referring to algorithms that do not explicitly use labeled STS data, reserving the term "unsupervised" for methods which are completely unsupervised/selfsupervised throughout their training process. We hope proceeding work will follow this convention in order to align the terminology with other fields of Machine Learning and NLP.

Introduction
Creating semantically coherent sentence representations is an important and long standing goal in NLP. The use of such sentence representations is ubiquitous, and include tasks such as sentiment analysis (Yu and Jiang, 2016), question answering (Yadav et al., 2019), machine translation (Wang et al., 2017), and natural language inference (Talman et al., 2019). The quality of sentence representations is measured either by their performance in downstream tasks, or by their correspondence to human semantic similarity judgements (Conneau and Kiela, 2018a). This latter task is referred to as Semantic Textual Similarity (STS), and has become the standard way to benchmark the quality of sentence representations. This paper investigates the combination of recent SOTA methods for unsupervised and zeroshot STS. In addition to this, we introduce a simple, but novel, method for knowledge distillation when the aim is to learn sentence embeddings. Our method relies on the realisation that averaging the actual embeddings of multiple teacher models makes for good student regression targets. We evaluate our method on STS and find our method to be robust and work well with already established methods.
Additionally, we observe that previous supervised embedding methods are hampered by poor hyperparameter tuning. By performing a simple grid-search over the possible relative regression targets proposed by Carlsson et al. (2021), certain models see a significant performance increase. This results in a new SOTA for supervised STS.
Our top performing models are available at: Examples of such methods include Skip-Thoughts of Kiros et al. (2015), which trains an encoder-decoder to reconstruct surrounding sentences given an encoded passage. Logeswaran and Lee (2018) propose Quick-Thoughts, which instead formulates the training objective as a sentence context classification task. Li et al. (2020) propose to use Normalizing Flows to transform a pre-trained model's embedding space into a Gaussian Distribution. Finally, Carlsson et al. (2021) introduce Contrastive Tension, which tunes two independent models to yield similar embeddings for identical sentences, and dissimilar embeddings for non-identical sentences.
This paper investigates in particular the combinations of the supervised Natural Language Inference (NLI) task of Reimers and Gurevych (2019), the unsupervised Gaussian Flow Normalization (Flow) of Li et al. (2020), and the Self-Supervised Contrastive Tension (CT) algorithm of Carlsson et al. (2021).

Knowledge Distillation
Knowledge distillation is a method where instead of training a model directly towards a specific task, it is trained towards the information of either a strong teacher model, or an ensemble of teacher models (Hinton et al., 2015). Barkan et al. (2019) use this technique to learn an embedding space by training it to contain the same pairwise relationships between sentences as that of a finetuned Transformer model fine-tuned on sentence-pair tasks. Reimers and Gurevych (2020) distill cross-lingual sentence embeddings using parallel corpora, by directly approximating the embedding space of a English S-BERT fine-tuned on labeled STS data.
Our distillation scheme is mostly similar to that of Reimers and Gurevych (2020), in the sense that we directly train a model's embedding space to approximate an already existing space. However, we distill from an ensemble of models within the same language, by averaging the ensemble model's embedding space.

Sentence Ensemble Distillation
Our Sentence Ensemble Distillation (SED) objective aims to train a new model, which we refer to as "the distillation learner", by distilling the information from multiple already existing sentence embedding models. This is realised by training the distillation learner to minimize the Mean Squared Error (MSE) between its embeddings and the average sentence embedding from that of the ensemble models. Our SED objective only updates the weights of the distillation learner.
Formally, we describe our SED objective in the following terms. Using a set of N sentence embedding models E = {M 0 , M 1 , M 2 , ...M N }, where M i (S) produces a vector with D dimensions, given sentence S. The mean ensemble embedding S E is calculated as: The per-example loss for distillation learner f is then calculated as: where f (S) also produces a vector with D dimensions.

Setup
Both distillation learners and ensemble models are pre-trained Transformers and always share the same architecture. For brevity, we limit this to BERT-Large, with the exception of the Supervised STS tasks where we also include RoBERTa-Large (Liu et al., 2020). Additional results for BERT-Distill and BERT-Base are available in Appendix-A.1. During SED we generate sentence embeddings from both the ensemble and the distillation learner by mean pooling the final layer. When evaluating the distillation learner, we mean pool over the two final layers (See Appendix-A.3 for a layerpooling study).
The fine-tuning objectives and their combinations are applied to each ensemble model sequentially, where the model name describes the order. However, the Flow normalization is only applied to the final distillation learner, and not to the ensemble models. For example, BERT-L-NLI-CT-SED-Flow is a Flow normalized BERT-Large model trained with SED from an ensemble of BERT-Large models, which each have been first fine-tuned towards the siamese NLI task and then tuned with CT.
All SED experiments use the same set of hyperparameters, with the same set of 100k randomly sampled sentences from the English Wikipedia and always using an ensemble of 10 models. The distillation is performed with the Adam optimizer using a batch size of 32, a final learning rate of 2e −5 and a linear warm-up schedule for 10% of the training data. The hyperparameters for the separate fine-tuning tasks of NLI, CT and Flow are taken from their respective publications, and are listed in Appendix-B for completeness. Table 1 shows results for the various unsupervised methods and their combinations when evaluated on the zero-shot English STS tasks of Agirre et al. (2012,2013,2014,2015,2016). As the SED itself requires no labeled data, the distillation learner is deemed unsupervised if its ensemble is trained without labeled data. Hence, we utilize both Flow and CT for these ensemble models, but not the siamese NLI objective.

Unsupervised STS
We observe that the combination of previously proposed methods lead to improved performance, and that adding the SED method further increases the performance. We set a new SOTA for unsupervised STS with an average score of 77.75 (Pearson) and 77.15 (Spearman). We also note a sometimes significant discrepancy between previously reported results and results replicated in our experiments. Table 2 shows results for the various methods which require labeled data when evaluated on the English zero-shot STS tasks. We observe a similiar tendency as in section-5.2, combining previous approaches leads to improved performance. However, the addition of SED does not contribute as much for the zero-shot methods. Again, we note discrepancies between previously reported results and our own.

Distillation Stability
In order to investigate the stability of the SED method, we train 10 distillation learners for each ensemble and compare their final performance on the zero-shot STS tasks. Table 3 reports the max, mean and standard deviation for the average Spearman score on the zero-shot STS task, for each model in the different ensembles, and for their respective 10 distillation learners. Finally, we report the results attained when utilizing the full ensemble for prediction. The average case of the SED models all outperform the best performing model of their respective ensemble. The standard deviation between the performance of different runs of SED is significantly smaller than that of the models within the ensembles.   Similar to the findings of Carlsson et al. (2021), who found no improvements to the supervised task with either NLI or CT, we see no improvements in Table 4 of either SED or Flow. However, the grid hyperparameter search over the regression lower bounds results in significant improvements for RoBERTa-large, while having no noticeable effect on BERT-Large. The improvements seen with RoBERTa-Large result in a new supervised STS SOTA, of 87.56 (Pearson) and 88.50 (Spearman).

Conclusion
There have been several recent suggestions on how to improve the unsupervised ability of Transformer language models to quantify semantic textual similarity. This paper analyzed the possibility to combine several of these proposed methods, and introduced a Sentence Ensemble Distillation (SED) method that trains a distillation learner to approximate the average embedding space of multiple existing embedding models. Combining previously proposed methods and our distillation objective, we set a new SOTA for unsupervised STS with an average of circa 2 Spearman points. We also contribute a new Supervised SOTA (with an improvement of circa 2 Spearman points) by proper hyperparameter tuning. Interestingly, we see no improvement for su-pervised STS when incorporating either previous methods or the SED objective. However, we note that the difference between the top unsupervised and the top supervised methods is only 6.45 (Spearman), and that the difference between our best fully unsupervised method and our best method using NLI data is only 1.17 (Spearman) on the zero-shot task.

A.1 Results with BERT-Distill & BERT-Base
We perform identical experiments to the unsupervised STS in section-5.2 and the zero-shot STS experiments of section-5.3, but also including BERT-Distill and BERT-Base. Table-5 shows the results for the methods which don't require any labeled data, and Table-6 shows the results for methods relying on labeled NLI data.
In the unsupervised STS tasks we see that BERT-Base performs the worse out of the three models, with BERT-Large performing 2.54 Spearman points better than BERT-Distill. The difference between the three models are smaller when using NLI data, where BERT-Large is the best, followed by BERT-Base and finally BERT-Distill. The difference between the best BERT-Large model and the BERT-Distill model is less than 2 Spearman points. Interestingly, both BERT-Base and BERT-Distill react negatively to being Flow Normalized after SED, and their best combinations seems to be NLI-CT-SED.

A.2 SentEval
Although the SentEval test bed of Conneau and Kiela (2018b), has fallen a bit out of fashion as recent publications have been more focused towards STS, Table-7 includes results for all its downstream tasks which are not the zero-shot STS tasks. The combination of NLI-CT-SED performs the best in most tasks and also in the average score. However, this improvement is quite marginal for most tasks. We find the Flow normalization to hamper performance on all tasks, sometimes significantly.

A.3 Layer Pooling Ablation
The results in Table-8 shows an layer-pooling ablation study for mean zero-shot STS Spearman correlation. Pooling from the two final layers is almost always preferable to only using a single layer. This goes for both fine-tuning objectives using NLI data and the combinations of self-supervised objectives. However, the differences between two and three layers are more sporadic. The best selfsupervised method is found using two layers, and the best NLI score is found pooling from three layers.

B Hyperparameters
B.1 Siamese Natural Language Inference (NLI) All NLI models are trained according to the parameters and settings given in the original paper of Reimers and Gurevych (2019). Additionally, the code used for all NLI models is from the official S-BERT Github 1 . The labeled data used is the combined datasets of SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018), which results in 1 million labeled training examples. We perform one epoch over the data using the Adam Optimizer, a learning rate of 2e −5 and a linear warm-up schedule over 10% of the data.

B.2 Contrastive Tension (CT)
All CT models are trained according to the parameters given in the original paper of Carlsson et al. (2021). Additionally, the code used for all CT models is from the official CT Github: github.com/FreddeFrallan/Contrastive-Tension. Using the RMSProp optimizer (Hinton, 2012) with a fixed learning rate schedule that decreases from 1e −5 to 2e −6 , we perform 50,000 update steps. Sentences randomly sampled from the English Wikipedia 2 , where we have 7 nonidentical sentence pairs for each positive sentence pair and a batch of 16. Hence, every batch consists of 2 positive sentence pairs and 14 negative sentence pairs.

B.3 Normalizing Flows (Flow)
All Flow normalization are trained according to the parameters given in the original paper of Li et al. (2020). Additionally, the code used for all Flow models is from the official BERT-Flow Github 3 . The Flow normalization is always done in regards to the combined data from all zero-shot STS tasks, with a learning rate of 1e −3 and 1 epoch. The coupling architecture used is described in Table-7 of the original paper of (Li et al., 2020).