We propose a unified framework for building unsupervised representations of individual objects or entities (and their compositions), by associating with each object both a distributional as well as a point estimate (vector embedding). This is made possible by the use of optimal transport, which allows us to build these associated estimates while harnessing the underlying geometry of the ground space. Our method gives a novel perspective for building rich and powerful feature representations that simultaneously capture uncertainty (via a distributional estimate) and interpretability (with the optimal transport map). As a guiding example, we formulate unsupervised representations for text, in particular for sentence representation and entailment detection. Empirical results show strong advantages gained through the proposed framework. This approach can be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data. The key tools underlying the framework are Wasserstein distances and Wasserstein barycenters (and, hence the title!). One of the main driving factors behind the recent surge of interest and successes in natural language processing and machine learning has been the development of better representation methods for data modalities. Examples include continuous vector representations for language (Mikolov et al., 2013; Pennington et al., 2014) , convolutional neural network (CNN) based text representations (Kim, 2014; Kalchbrenner et al., 2014; Severyn and Moschitti, 2015; BID4 , or via other neural architectures such as RNNs, LSTMs BID14 Collobert and Weston, 1 And, hence the title! 2008), all sharing one core idea -to map input entities to dense vector embeddings lying in a lowdimensional latent space where the semantics of the inputs are preserved.While existing methods represent each entity of interest (e.g., a word) as a single point in space (e.g., its embedding vector), we here propose a fundamentally different approach. We represent each entity based on the histogram of contexts (cooccurring with it), with the contexts themselves being points in a suitable metric space. This allows us to cast the distance between histograms associated with the entities as an instance of the optimal transport problem (Monge, 1781; Kantorovich, 1942; Villani, 2008) . For example, in the case of words as entities, the resulting framework then intuitively seeks to minimize the cost of moving the set of contexts of a given word to the contexts of another. Note that the contexts here can be words, phrases, sentences, or general entities cooccurring with our objects to be represented, and these objects further could be any type of events extracted from sequence data, including e.g., products such as movies or web-advertisements BID8 , nodes in a graph BID9 , or other entities (Wu et al., 2017) . Any co-occurrence structure will allow the construction of the histogram information, which is the crucial building block for our approach.A strong motivation for our proposed approach here comes from the domain of natural language, where the entities (words, phrases or sentences) generally have multiple semantics under which they are present. Hence, it is important that we consider representations that are able to effectively capture such inherent uncertainty and polysemy, and we will argue that histograms (or probability distributions) over embeddings allows to capture more of this information compared to point-wise embeddings alone. We will call the histogram as the distributional estimate of our object of interest, while we refer to the individual embeddings of single contexts as point estimates.Next, for the sake of clarity, we discuss the framework in the concrete use-case of text representations, when the contexts are just words, by employing the well-known Positive Pointwise Mutual Information (PPMI) matrix to compute the histogram information for each word.With the power of optimal transport, we show how this framework can be of significant use for a wide variety of important tasks in NLP, including word and sentence representations as well as hypernymy (entailment) detection, and can be readily employed on top of existing pre-trained embeddings for the contexts. The connection to optimal transport at the level of words and contexts paves the way to make better use of its vast toolkit (like Wasserstein distances, barycenters, etc.) for applications in NLP, which in the past has primarily been restricted to document distances (Kusner et al., 2015; BID16 .We demonstrate that building the required histograms comes at almost no additional cost, as the co-occurrence counts are obtained in a single pass over the corpus. Thanks to the entropic regularization introduced by Cuturi (2013), Optimal Transport distances can be computed efficiently in a parallel and batched manner on GPUs. Lastly , the obtained transport map FIG0 ) also provides for interpretability of the suggested framework. To sum up, we advocate for associating both a distributional and point estimate as a representation for each entity. We show how this allows us to use optimal transport over the set of contexts associated with these entities, in problems with a co-occurrence structure. Further, the framework Aitor Gonzalez-Agirre. 2012. Semeval-2012 In particular, when β = 1, we recover the equation for histograms as in Section 5, and β = 0 would imply normalization with respect to cluster sizes.
Represent each entity based on its histogram of contexts and then Wasserstein is all you need!
{"src": "scitldr-AIC_train", "paper_id": "Bkx2jd4Nx7"}