Most deep learning for NLP represents each word with a single point or single-mode region in semantic space, while the existing multi-mode word embeddings cannot represent longer word sequences like phrases or sentences. We introduce a phrase representation (also applicable to sentences) where each phrase has a distinct set of multi-mode codebook embeddings to capture different semantic facets of the phrase's meaning. The codebook embeddings can be viewed as the cluster centers which summarize the distribution of possibly co-occurring words in a pre-trained word embedding space. We propose an end-to-end trainable neural model that directly predicts the set of cluster centers from the input text sequence (e.g., a phrase or a sentence) during test time. We find that the per-phrase/sentence codebook embeddings not only provide a more interpretable semantic representation but also outperform strong baselines (by a large margin in some tasks) on benchmark datasets for unsupervised phrase similarity, sentence similarity, hypernym detection, and extractive summarization. Many widely-applicable NLP models learn a representation from only co-occurrence statistics in the raw text without any supervision. Examples include word embedding like word2vec (Mikolov et al., 2013) or GloVe (Pennington et al., 2014) , sentence embeddings like skip-thoughts (Kiros et al., 2015) , and contextualized word embedding like ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) . Most of these models use a single embedding to represent one sentence or one phrase and can only provide symmetric similarity measurement when no annotation is available. However, a word or phrase might have multiple senses, and a sentence can involve multiple topics, which are hard to analyze based on a single embedding without supervision. To address the issue, word sense induction methods (Lau et al., 2012) and recent multi-mode word embeddings (Neelakantan et al., 2014; Athiwaratkun & Wilson, 2017; Singh et al., 2018) represent each target word as multiple points or regions in a distributional semantic space by (explicitly or implicitly) clustering all the words appearing beside the target word. In Figure 1 , the multi-mode representation of real property is illustrated as an example. Real property can be observed in legal documents where it usually means a real estate, while a real property can also mean a true characteristic in philosophic discussions. The previous approaches discover those senses by clustering observed neighboring words (e.g., company and tax). In contrast with topic modeling like LDA (Blei et al., 2003) , the approaches need to solve a distinct clustering problem for every target word while topic modeling finds a single set of clusters by clustering all the words in the corpus. Extending these multi-mode representations to arbitrary sequences like phrases or sentences is difficult due to two efficiency challenges. First, there are usually many more unique phrases and sentences in a corpus than there are words, while the number of parameters for clustering-based approaches is O(|V | × |K| × |E|), where |V | is number of unique sequences, |K| is number of modes/clusters, and |E| is the number of embedding dimensions. Estimating and storing such a large number of parameters take time and space. More important, many unique sequences imply much fewer co-occurring words to be clustered for each sequence, especially for long sequences Figure 1 : The target phrase real property is represented by four clustering centers. The previous work discovers the four modes by finding clustering centers which well compress the embedding of observed co-occurring words. Instead, our compositional model learns to predict the embeddings of cluster centers from the sequence of words in the target phrase so as to reconstruct the (unseen) co-occurring distribution well. like sentences, so an effective model needs to overcome this sample efficient challenge (i.e., sparseness in the co-occurring statistics). However, clustering approaches often have too many parameters to learn the compositional meaning of each sequence without overfitting. Nevertheless, the sentences (or phrases) sharing multiple words tend to have similar cluster centers, so we should be able to compress many redundant parameters in these local clustering problems to circumvent the challenges. In this work, we adopt a neural encoder and decoder to achieve the goal. As shown in Figure 1 , instead of clustering co-occurring words beside a target sequence at test time as in previous approaches, we learn a mapping between the target sequence (i.e., phrases or sentences) and the corresponding cluster centers during training so that we can directly predict those cluster centers using a single forward pass of the neural network for an arbitrary unseen input sequences during testing. To allow the neural network to generate the cluster centers in an arbitrary order, we use a nonnegative and sparse coefficient matrix to dynamically match the sequence of predicted cluster centers and the observed set of co-occurring word embeddings during training. After the coefficient matrix is estimated for each input sequence, the gradients are back-propagated to cluster centers (i.e., codebook embeddings) and weights of decoder and encoder, which allows us to train the whole model jointly and end-to-end. In experiments, we show that the proposed model captures the compositional meanings of words in unsupervised phrase similarity tasks much better than averaging their (contextualized) word embeddings, strong baselines that are widely used in practice. In addition to similarity, our model can also measure asymmetric relations like hypernymy without any supervision. Furthermore, the multimode representation is shown to outperform the single-mode alternatives in sentence representation, especially as demonstrated in our extractive summarization experiment. In this work, we overcome the computational and sampling efficiency challenges of learning the multi-mode representation for long sequences like phrases or sentences. We use a neural encoder to model the compositional meaning of the target sequence and use a neural decoder to predict a set of codebook embeddings as the representation of the sentences or phrases. During training, we use a non-negative sparse coefficient matrix to dynamically match the predicted codebook embeddings to a set of observed co-occurring words and allow the neural decoder to predict the clustering centers with an arbitrary permutation. We demonstrate that the proposed models can learn to predict interpretable clustering centers conditioned on an (unseen) sequence, and the representation outperforms widely-used baselines such as BERT, skip-thoughts and various approaches based on GloVe in several unsupervised benchmarks. The experimental results also suggest that multi-facet embeddings perform the best when the input sequence (e.g., a sentence) involves many aspects, while multi-facet and single-facet embeddings perform similarly good when the input sequence (e.g., a phrase) usually involves only one aspect. In the future, we would like to train a single model which could generate multi-facet embeddings for both phrases and sentences, and evaluate the method as a pre-trained embedding approach for supervised or semi-supervised settings. Furthermore, we plan to apply this method to other unsupervised learning tasks that heavily rely on co-occurrence statistics such as graph embedding or recommendation.
We propose an unsupervised way to learn multiple embeddings for sentences and phrases 
{"src": "scitldr-AIC_train", "paper_id": "HkebMlrFPS"}