Multi-view learning can provide self-supervision when different views are available of the same data. Distributional hypothesis provides another form of useful self-supervision from adjacent sentences which are plentiful in large unlabelled corpora. Motivated by the asymmetry in the two hemispheres of the human brain as well as the observation that different learning architectures tend to emphasise different aspects of sentence meaning, we present two multi-view frameworks for learning sentence representations in an unsupervised fashion. One framework uses a generative objective and the other a discriminative one. In both frameworks, the final representation is an ensemble of two views, in which, one view encodes the input sentence with a Recurrent Neural Network (RNN), and the other view encodes it with a simple linear model. We show that, after learning, the vectors produced by our multi-view frameworks provide improved representations over their single-view learnt counterparts, and the combination of different views gives representational improvement over each view and demonstrates solid transferability on standard downstream tasks. Multi-view learning methods provide the ability to extract information from different views of the data and enable self-supervised learning of useful features for future prediction when annotated data is not available BID16 . Minimising the disagreement among multiple views helps the model to learn rich feature representations of the data and, also after training, the ensemble of the feature vectors from multiple views can provide an even stronger generalisation ability.Distributional hypothesis BID22 noted that words that occur in similar contexts tend to have similar meaning BID51 , and distributional similarity BID19 consolidated this idea by stating that the meaning of a word can be determined by the company it has. The hypothesis has been widely used in machine learning community to learn vector representations of human languages. Models built upon distributional similarity don't explicitly require human-annotated training data; the supervision comes from the semantic continuity of the language data.Large quantities of annotated data are usually hard and costly to obtain, thus it is important to study unsupervised and self-supervised learning. Our goal is to propose learning algorithms built upon the ideas of multi-view learning and distributional hypothesis to learn from unlabelled data. We draw inspiration from the lateralisation and asymmetry in information processing of the two hemispheres of the human brain where, for most adults, sequential processing dominates the left hemisphere, and the right hemisphere has a focus on parallel processing BID9 , but both hemispheres have been shown to have roles in literal and non-literal language comprehension BID14 BID15 .Our proposed multi-view frameworks aim to leverage the functionality of both RNN-based models, which have been widely applied in sentiment analysis tasks BID57 , and the linear/log-linear models, which have excelled at capturing attributional similarities of words and sentences BID5 BID6 BID24 BID51 for learning sentence representations. Previous work on unsupervised sentence representation learning based on distributional hypothesis can be roughly categorised into two types:Generative objective: These models generally follow the encoder-decoder structure. The encoder learns to produce a vector representation for the current input, and the decoder learns to generate sentences in the adjacent context given the produced vector BID28 BID24 BID20 BID50 . The idea is straightforward, yet its scalability for very large corpora is hindered by the slow decoding process that dominates training time, and also the decoder in each model is discarded after learning as the quality of generated sequences is not the main concern, which is a waste of parameters and learning effort.Our first multi-view framework has a generative objective and uses an RNN as the encoder and an invertible linear projection as the decoder. The training time is drastically reduced as the decoder is simple, and the decoder is also utilised after learning. A regularisation is applied on the linear decoder to enforce invertibility, so that after learning, the inverse of the decoder can be applied as a linear encoder in addition to the RNN encoder.Discriminative Objective: In these models, a classifier is learnt on top of the encoders to distinguish adjacent sentences from those that are not BID31 BID26 BID40 BID33 ; these models make a prediction using a predefined differential similarity function on the representations of the input sentence pairs or triplets.Our second multi-view framework has a discriminative objective and uses an RNN encoder and a linear encoder; it learns to maximise agreement among adjacent sentences. Compared to earlier work on multi-view learning BID16 BID17 BID52 that takes data from various sources or splits data into disjoint populations, our framework processes the exact same data in two distinctive ways. The two distinctive information processing views tend to encode different aspects of an input sentence; forcing agreement/alignment between these views encourages each view to be a better representation, and is beneficial to the future use of the learnt representations.Our contribution is threefold:• Two multi-view frameworks for learning sentence representations are proposed, in which one framework uses a generative objective and the other one adopts a discriminative objective. Two encoding functions , an RNN and a linear model, are learnt in both frameworks.• The results show that in both frameworks, aligning representations from two views gives improved performance of each individual view on all evaluation tasks compared to their single-view trained counterparts, and furthermore ensures that the ensemble of two views provides even better results than each improved view alone.• Models trained under our proposed frameworks achieve good performance on the unsupervised tasks, and overall outperform existing unsupervised learning models, and armed with various pooling functions, they also show solid results on supervised tasks, which are either comparable to or better than those of the best unsupervised transfer model. It is shown BID24 that the consistency between supervised and unsupervised evaluation tasks is much lower than that within either supervised or unsupervised tasks alone and that a model that performs well on supervised tasks may fail on unsupervised tasks. BID13 subsequently showed that, with a labelled training corpus, such as SNLI BID8 and MultiNLI BID56 , the resulting representations of the sentences from the trained model excel in both supervised and unsupervised tasks. Multi-task learning BID48 also gives impressive performance on downstream tasks while labelled data is costly. Our model is able to achieve good results on both groups of tasks without labelled information. In both frameworks, RNN encoder and linear encoder perform well on all tasks, and generative objective and discriminative objective give similar performance. We proposed multi-view sentence representation learning frameworks with generative and discriminative objectives; each framework combines an RNN-based encoder and an average-on-word-vectors linear encoder and can be efficiently trained within a few hours on a large unlabelled corpus. The experiments were conducted on three large unlabelled corpora, and meaningful comparisons were made to demonstrate the generalisation ability and transferability of our learning frameworks and consolidate our claim. The produced sentence representations outperform existing unsupervised transfer methods on unsupervised evaluation tasks, and match the performance of the best unsupervised model on supervised evaluation tasks.Our experimental results support the finding BID24 that linear/log-linear models (g in our frameworks) tend to work better on the unsupervised tasks, while RNN-based models (f in our frameworks) generally perform better on the supervised tasks. As presented in our experiments, multi-view learning helps align f and g to produce better individual representations than when they are learned separately. In addition, the ensemble of both views leveraged the advantages of both, and provides rich semantic information of the input sentence. Future work should explore the impact of having various encoding architectures and learning under the multi-view framework.Our multi-view learning frameworks were inspired by the asymmetric information processing in the two hemispheres of the human brain, in which the left hemisphere is thought to emphasise sequential processing and the right one more parallel processing BID9 . Our experimental results raise an intriguing hypothesis about how these two types of information processing may complementarily help learning.
Multi-view learning improves unsupervised sentence representation learning
{"src": "scitldr-AIC_train", "paper_id": "HyxaHGcij7"}