We present a pairwise context-sensitive Autoencoder for computing text pair similarity . Our model encodes input text into context-sensitive representations and uses them to compute similarity between text pairs . Our model outperforms the state-of-the-art models in two semantic retrieval tasks and a contextual word similarity task . For retrieval , our unsupervised approach that merely ranks inputs with respect to the cosine similarity between their hidden representations shows comparable performance with the state-of-the-art supervised models and in some cases outperforms them . Representation learning algorithms learn representations that reveal intrinsic low-dimensional structure in data ( Bengio et al . , 2013 ) . Such representations can be used to induce similarity between textual contents by computing similarity between their respective vectors ( Huang et al . , 2012 ; Silberer and Lapata , 2014 ) . Recent research has made substantial progress on semantic similarity using neural networks ( Rothe and Schütze , 2015 ; Dos Santos et al . , 2015 ; Severyn and Moschitti , 2015 ) . In this work , we focus our attention on deep autoencoders and extend these models to integrate sentential or document context information about their inputs . We represent context information as low dimensional vectors that will be injected to deep autoencoders . To the best of our knowledge , this is the first work that enables integrating context into autoencoders . In representation learning , context may appear in various forms . For example , the context of a current sentence in a document could be either its neighboring sentences ( Lin et al . , 2015 ; Wang and Cho , 2015 ) , topics associated with the sentence ( Mikolov and Zweig , 2012 ; Le and Mikolov , 2014 ) , the document that contains the sentence ( Huang et al . , 2012 ) , as well as their combinations ( Ji et al . , 2016 ) . It is important to integrate context into neural networks because these models are often trained with only local information about their individual inputs . For example , recurrent and recursive neural networks only use local information about previously seen words in a sentence to predict the next word or composition . 1On the other hand , context information ( such as topical information ) often capture global information that can guide neural networks to generate more accurate representations . We investigate the utility of context information in three semantic similarity tasks : contextual word sense similarity in which we aim to predict semantic similarity between given word pairs in their sentential context ( Huang et al . , 2012 ; Rothe and Schütze , 2015 ) , question ranking in which we aim to retrieve semantically equivalent questions with respect to a given test question ( Dos Santos et al . , 2015 ) , and answer ranking in which we aim to rank single-sentence answers with respect to a given question ( Severyn and Moschitti , 2015 ) . The contributions of this paper are as follows : ( 1 ) integrating context information into deep autoencoders and ( 2 ) showing that such integration improves the representation performance of deep autoencoders across several different semantic similarity tasks . Our model outperforms the state-of-the-art su-pervised baselines in three semantic similarity tasks . Furthermore , the unsupervised version of our autoencoder show comparable performance with the supervised baseline models and in some cases outperforms them . 2 Context-sensitive Autoencoders We introduce an effective approach to integrate sentential or document context into deep autoencoders and show that such integration is important in semantic similarity tasks . In the future , we aim to investigate other types of linguistic context ( such as POS tag and word dependency information , word sense , and discourse relations ) and develop a unified representation learning framework that integrates such linguistic context with representation learning models .
Existing approaches for textual representation learning only use local information without contexts which capture global information that can guide neural networks in generating accurate representations.
They propose a pairwise context-sensitive Autoencoder which integrates sentential or document context for computing text pair similarity.
The proposed model outperforms the state-of-the-art models in two retrieval and word similarity tasks and an unsupervised version performs comparable with several supervised baselines.