Contrastive learning has achieved impressive success in generation tasks to militate the " exposure bias " problem and discriminatively exploit the different quality of references . Existing works mostly focus on contrastive learning on the instance-level without discriminating the contribution of each word , while keywords are the gist of the text and dominant the constrained mapping relationships . Hence , in this work , we propose a hierarchical contrastive learning mechanism , which can unify hybrid granularities semantic meaning in the input text . Concretely , we first propose a keyword graph via contrastive correlations of positive-negative pairs to iteratively polish the keyword representations . Then , we construct intra-contrasts within instance-level and keyword-level , where we assume words are sampled nodes from a sentence distribution . Finally , to bridge the gap between independent contrast levels and tackle the common contrast vanishing problem , we propose an inter-contrast mechanism that measures the discrepancy between contrastive keyword nodes respectively to the instance distribution . Experiments demonstrate that our model outperforms competitive baselines on paraphrasing , dialogue generation , and storytelling tasks . Generation tasks such as storytelling , paraphrasing , and dialogue generation aim at learning a certain correlation between text pairs that maps an arbitrary-length input to another arbitrary-length output . Traditional methods are mostly trained with " teacher forcing " and lead to an " exposure bias " problem ( Schmidt , 2019 ) . Incorporating the generation method with contrastive learning achieved impressive performance on tackling such issues , which takes an extra consideration of synthetic negative samples contrastively ( Lee et al . , 2021 Existing contrastive mechanisms are mainly focused on the instance level ( Lee et al . , 2021 ; Cai et al . , 2020 ) . However , word-level information is also of great importance . Take the case shown in the upper part of Figure 1 for example , the keyword covers the gist of the input text and determines the embedding space of the text . The text representation will be significantly affected if adding a slight perturbation on the keyword , i.e. , changing " cosmology " to " astrophysics " . In addition , as shown on the bottom part , under some circumstances , it is too easy for the model to do the classification since the semantic gap between contrastive pairs is huge . Thus , the model fails to distinguish the actual discrepancy , which causes a " contrast vanishing " problem at both instance-level and keyword-level . Based on the above motivation , in this paper , we propose a hierarchical contrastive learning method built on top of the classic CVAE structure . We choose CVAE due to its ability in modeling global properties such as syntactic , semantic , and discourse coherence ( Li et al . , 2015 ; Yu et al . , 2020 ) . We first learn different granularity representations through two independent contrast , i.e. , instancelevel and keyword-level . Specifically , we use the universal and classic TextRank ( Mihalcea and Tarau , 2004 ) method to extract keywords from each text , which contain the most important information and need to be highlighted . On the instancelevel , we treat the keyword in the input text as an additional condition for a better prior semantic distribution . Then , we utilize Kullback-Leibler divergence ( Kullback and Leibler , 1951 ) to reduce the distance between prior distribution and positive posterior distribution , and increase the distance with the negative posterior distribution . While on the keyword-level , we propose a keyword graph via contrastive correlations of positive-negative pairs to learn informative and accurate keyword representations . By treating the keyword in the output text as an anchor , the imposter keyword is produced by neighboring nodes of the anchor keyword and forms the keyword-level contrast , where the similarity between the imposter keyword and the anchor keyword is poorer than the positive keyword . To unify individual intra-contrasts and tackle the " contrast vanishing " problem in independent contrastive granularities , we leverage an inter-contrast , the Mahalanobis contrast , to investigate the contrastive enhancement based on the Mahalanobis distance ( De Maesschalck et al . , 2000 ) , a measure of the distance between a point and a distribution , between the instance distribution and the keyword representation . Concretely , we ensure the distance from the anchor instance distribution to the groundtruth keyword vector is closer than to the imposter keyword vector . The Mahalanobis contrast plays an intermediate role that joins the different granularities contrast via incorporating the distribution of instance with the representation of its crucial part , and makes up a more comprehensive keyworddriven hierarchical contrastive mechanism , so as to ameliorate the generated results . We empirically show that our model outperforms CVAE and other baselines significantly on three generation tasks : paraphrasing , dialogue genera-tion , and storytelling . Our contributions can be summarized as follows : • To our best knowledge , we are the first to propose an inter-level contrastive learning method , which unifies instance-level and keyword-level contrasts in the CVAE framework . • We propose three contrastive learning measurements : KL divergence for semantic distribution , cosine distance for points , and Mahalanobis distance for points with distribution . • We introduce a global keyword graph to obtain polished keyword representations and construct imposter keywords for contrastive learning . In this paper , we propose a hierarchical contrastive learning mechanism , which consists of intra-contrasts within instance-level and keywordlevel and inter-contrast with Mahalanobis contrast . The experimental results yield significant out-performance over baselines when applied in the CVAE framework . In the future , we aim to extend the contrastive learning mechanism to different basic models , and will explore contrastive learning methods based on external knowledge .
Existing works on contrastive learning for text generation focus only on instance-level while word-level information such as keywords is also of great importance.
They propose a CVAE-based hierarchical contrastive learning within instance and keyword-level using a keyword graph which iteratively polishes the keyword representations.
The proposed model outperforms CVAE and baselines on storytelling, paraphrasing, and dialogue generation tasks.