Learning sentence embeddings from dialogues has drawn increasing attention due to its low annotation cost and high domain adaptability . Conventional approaches employ the siamese-network for this task , which obtains the sentence embeddings through modeling the context-response semantic relevance by applying a feed-forward network on top of the sentence encoders . However , as the semantic textual similarity is commonly measured through the element-wise distance metrics ( e.g. cosine and L2 distance ) , such architecture yields a large gap between training and evaluating . In this paper , we propose DialogueCSE , a dialogue-based contrastive learning approach to tackle this issue . DialogueCSE first introduces a novel matching-guided embedding ( MGE ) mechanism , which generates a contextaware embedding for each candidate response embedding ( i.e. the context-free embedding ) according to the guidance of the multi-turn context-response matching matrices . Then it pairs each context-aware embedding with its corresponding context-free embedding and finally minimizes the contrastive loss across all pairs . We evaluate our model on three multi-turn dialogue datasets : the Microsoft Dialogue Corpus , the Jing Dong Dialogue Corpus , and the E-commerce Dialogue Corpus . Evaluation results show that our approach significantly outperforms the baselines across all three datasets in terms of MAP and Spearman 's correlation measures , demonstrating its effectiveness . Further quantitative experiments show that our approach achieves better performance when leveraging more dialogue context and remains robust when less training data is provided . Sentence embeddings are used with success for a variety of NLP applications ( Cer et al . , 2018 ) and many prior methods have been proposed with different learning schemes . Kiros et al . ( 2015 ) ; Logeswaran and Lee ( 2018 ) ; Hill et al . ( 2016 ) train sentence encoders in a self-supervised manner with web pages and books . Conneau et al . ( 2017 ) ; Cer et al . ( 2018 ) ; Reimers and Gurevych ( 2019 ) propose to learn sentence embeddings on the supervised datasets such as SNLI ( Bowman et al . , 2015 ) and MNLI ( Williams et al . , 2018 ) . Although the supervised-learning approaches achieve better performance , they suffer from high cost of annotation in building the training dataset , which makes them hard to adapt to other domains or languages . Recently , learning sentence embeddings from dialogues has begun to attract increasing attention . Dialogues provide strong semantic relationships among conversational utterances and are usually easy to collect in large amounts . Such advantages make the dialogue-based self-supervised learning methods promising to achieve competitive or even superior performance against the supervised-learning methods , especially under the low-resource conditions . While promising , the issue of how to effectively exploit the dialogues for this task has not been sufficiently explored . Yang et al . ( 2018 ) propose to train an input-response prediction model on Reddit dataset ( Al-Rfou et al . , 2016 ) . Since they build their architecture based on the single-turn dialogue , the multi-turn dialogue history is not fully exploited . Henderson et al . ( 2020 ) demonstrate that introducing the multi-turn dialogue context can improve the sentence embedding performance . However , they concatenate the multi-turn dialogue context into a long token sequence , failing to model intersentence semantic relationships among the utterances . Recently , more advanced methods such as ( Reimers and Gurevych , 2019 ) achieve better performance by employing BERT ( Devlin et al . , 2019 ) as the sentence encoder . These works have in common that they employ a feed-forward network with a non-linear activation on top of the sentence en-coders to model the context-response semantic relevance , thereby learning the sentence embeddings . However , such architecture presents two limitations : ( 1 ) It yields a large gap between training and evaluating , since the semantic textual similarity is commonly measured by the element-wise distance metrics such as cosine and L2 distance . ( 2 ) Concatenating all the utterances in the dialogue context inevitably introduces the noise as well as the redundant information , resulting in a poor result . In this paper , we propose DialogueCSE , a dialogue-based contrastive learning approach to tackle these issues . We hold that the semantic matching relationships between the context and the response can be implicitly modeled through contrastive learning , thus making it possible to eliminate the gap between training and evaluating . To this end , we introduce a novel matching-guided embedding ( MGE ) mechanism . Specifically , MGE first pairs each utterance in the context with the response and performs a token-level dot-product operation across all the utterance-response pairs to obtain the multi-turn matching matrices . Then the multi-turn matching matrices are used as guidance to generate a context-aware embedding for the response embedding ( i.e. the context-free embedding ) . Finally , the context-aware embedding and the context-free embedding are paired as a training sample , whose label is determined by whether the context and the response are originally from the same dialogue . Our motivation is that once the context semantically matches the response , it has the ability to distill the context-aware information from the context-free embedding , which is exactly the learning objective of the sentence encoder that aims to produce context-aware sentence embeddings . We train our model on three multi-turn dialogue datasets : the Microsoft Dialogue Corpus ( MDC ) ( Li et al . , 2018 ) , the Jing Dong Dialogue Corpus ( JDDC ) ( Chen et al . , 2020 ) , and the E-commerce Dialogue Corpus ( ECD ) ( Zhang et al . , 2018 ) . To evaluate our model , we introduce two types of tasks : the semantic retrieval ( SR ) task and the dialogue-based semantic textual similarity ( D-STS ) task . Here we do not adopt the standard semantic textual similarity ( STS ) task ( Cer et al . , 2017 ) for two reasons : ( 1 ) As revealed in ( Zhang et al . , 2020 ) , the sentence embedding performance varies greatly as the domain of the training data changes . As a dialogue dataset is always about several certain domains , evaluating on the STS benchmark may mis-lead the evaluation of the model . ( 2 ) The dialoguebased sentence embeddings focus on context-aware rather than context-free semantic meanings , which may not be suitable to be evaluated through the context-free benchmarks . Since previous dialoguebased works have not set up a uniform benchmark , we construct two evaluation datasets for each dialogue corpus . A total of 18,964 retrieval samples and 4,000 sentence pairs are annotated by seven native speakers through the crowd-sourcing platform1 . The evaluation results indicate that DialogueCSE significantly outperforms the baselines on the three datasets in terms of both MAP and Spearman 's correlation metrics , demonstrating its effectiveness . Further quantitative experiments show that Dia-logueCSE achieves better performance when leveraging more dialogue context and remains robust when less training data is provided . To sum up , our contributions are threefold : • We propose DialogueCSE , a dialogue-based contrastive learning approach with MGE mechanism for learning sentence embeddings from dialogues . As far as we know , this is the first attempt to apply contrastive learning in this area . • We construct the dialogue-based sentence embedding evaluation benchmarks for three dialogue corpus . All of the datasets will be released to facilitate the follow-up researches . • Extensive experiments show that Dia-logueCSE significantly outperforms the baselines , establishing the state-of-the-art results . 2 Related Work In this work , we propose DialogueCSE , a dialoguebased contrastive learning approach to learn sentence embeddings from dialogues . We also propose uniform evaluation benchmarks for evaluating the quality of the dialogue-based sentence embeddings . Evaluation results show that DialogueCSE achieves the best result over the baselines while adding no additional parameters . In the next step , we will study how to introduce more interaction information to learn the sentence embeddings and try to incorporate the contrast learning method into the pre-training stage .
Existing methods to learn representations from dialogues have a similarity-measurement gap between training and evaluation time and do not exploit the multi-turn structure of data.
They propose a dialogue-based contrastive learning approach to learn sentence embeddings from dialogues by modelling semantic matching relationships between the context and response implicitly.
The proposed approach outperforms baseline methods on two newly introduced tasks coupled with three multi-turn dialogue datasets in terms of MAP and Spearman's correlation measures.