A Survey of Sentiment Analysis Based on Transfer Learning <s> 1) PARAMETER-TRANSFER METHODS <s> The exponential increase in the availability of online reviews and recommendations makes sentiment classification an interesting topic in academic and industrial research. Reviews can span so many different domains that it is difficult to gather annotated training data for all of them. Hence, this paper studies the problem of domain adaptation for sentiment classifiers, hereby a system is trained on labeled reviews from one source domain but is meant to be deployed on another. We propose a deep learning approach which learns to extract a meaningful representation for each review in an unsupervised fashion. Sentiment classifiers trained with this high-level feature representation clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products. Furthermore, this method scales well and allowed us to successfully perform domain adaptation on a larger industrial-strength dataset of 22 domains. <s> BIB001 </s> A Survey of Sentiment Analysis Based on Transfer Learning <s> 1) PARAMETER-TRANSFER METHODS <s> Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters--in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB™, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks. <s> BIB002 </s> A Survey of Sentiment Analysis Based on Transfer Learning <s> 1) PARAMETER-TRANSFER METHODS <s> We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. <s> BIB003 </s> A Survey of Sentiment Analysis Based on Transfer Learning <s> 1) PARAMETER-TRANSFER METHODS <s> Deep learning, as a new unsupervised leaning algorithm, has strong capabilities to learn data representations. Previous work has shown that new features learned by deep learning algorithm help to improve the accuracy of cross-domain classification. In this paper, we firstly propose a modified version of marginalized stacked denoising autoencoders (mSDA). We call it mSDA++ algorithm, which can learn excellent and low-dimensional features for training classifier. In addition, we combine mSDA with EASYADAPT algorithm to further improve the accuracy of cross-domain classification. Then we use SVM, mSDA, mSDA++, and EA+mSDA algorithms to do the cross-domain sentiment classification experiments on Amazon benchmark dataset. The results show that EA+mSDA algorithm attains the best accuracy. Besides, the mSDA++ algorithm can accelerate the subsequent calculation and reduce the data storage space. <s> BIB004 </s> A Survey of Sentiment Analysis Based on Transfer Learning <s> 1) PARAMETER-TRANSFER METHODS <s> Computer vision has benefited from initializing multiple deep layers with weights pretrained on large supervised training sets like ImageNet. Natural language processing (NLP) typically sees initialization of only the lowest layer of deep models with pretrained word vectors. In this paper, we use a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation (MT) to contextualize word vectors. We show that adding these context vectors (CoVe) improves performance over using only unsupervised word and character vectors on a wide variety of common NLP tasks: sentiment analysis (SST, IMDb), question classification (TREC), entailment (SNLI), and question answering (SQuAD). For fine-grained sentiment analysis and entailment, CoVe improves performance of our baseline models to the state of the art. <s> BIB005 </s> A Survey of Sentiment Analysis Based on Transfer Learning <s> 1) PARAMETER-TRANSFER METHODS <s> We introduce a new type of deep contextualized word representation that models both (1) complex characteristics of word use (e.g., syntax and semantics), and (2) how these uses vary across linguistic contexts (i.e., to model polysemy). Our word vectors are learned functions of the internal states of a deep bidirectional language model (biLM), which is pre-trained on a large text corpus. We show that these representations can be easily added to existing models and significantly improve the state of the art across six challenging NLP problems, including question answering, textual entailment and sentiment analysis. We also present an analysis showing that exposing the deep internals of the pre-trained network is crucial, allowing downstream models to mix different types of semi-supervision signals. <s> BIB006 </s> A Survey of Sentiment Analysis Based on Transfer Learning <s> 1) PARAMETER-TRANSFER METHODS <s> Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code. <s> BIB007
Parameter-transfer method leverages the parameter sharing model of the source and target domains. Using the parametertransfer method, scholars can transfer the trained model parameters in a large number of datasets to the target task. In the parameter-transfer method applied to the NLP field, word2vec BIB003 which is a simple transfer technology only for the first layer of the model has many applications, which has a great impact in practice and be used in many advanced technology. Glorot et al. BIB001 studied the domain adaptive problem of sentiment classifiers, using Stacked Denoising Autoencoder (SDA) to solve edge distribution differences. The first step of the algorithm is to find an invariant potential feature space by the input space that has been trained by the Stacked Denoising Auto-encoder, and then uses the potential features and the labeled source data to train the classifier. This method is excellent in comment sentiment analysis, but it is more dependent on parameter initialization. The marginalized SDA (mSDA) BIB002 algorithm, which improved the shortcomings of the SDA algorithm: high computational cost and lack of scalability of high-dimensional features. The algorithm has powerful feature learning capabilities and does not require optimization algorithms to train parameters. Sun BIB004 proposed a modified version of the mSDA ++ algorithm for the marginalized SDA (mSDA), which can learn low-dimensional features. The combination of mSDA and EASYADAPT algorithm can improve the accuracy of text classification. In addition, the mSDA ++ algorithm accelerates subsequent calculations and reduces data storage space. Mccann et al. BIB005 and Peters et al. BIB006 used a method of combining embedding from other tasks with different levels of input. Mccann et al. BIB005 trained the neural network model through an English-German translation task to obtain an output called ''context vectors (CoVe)''. They used CoVe for many NLP tasks, including semantic sentiment analysis, problem classification, text implied, question and answer. Figure 2 is a schematic diagram of the process about combining other tasks with the target task. Peters et al. BIB006 proposed EMLo (Embeddings from Language Models), which is a method for extracting word vectors of deep semantic features. It can train the bidirectional LSTM model (biLM) on the big corpus at first, then connect its own neural network directly on the basis of biLM, take the output of biLM as the input of its own network, and use LSTM to generate characterization of words. In this way, a richer word representation can be introduced and applied to many types of NLP tasks, but this method still trains the main task model from the beginning, and regards pre-training embedding as a fixed parameter, limiting its effect. Dai and Le  first put forward a fine-tuning of the language model (LM) and proposed two methods for improving sequence learning with recurrent networks using unlabeled data.The function of the first method is to predict the subsequent content in the sequence; the second method can use the sequence-autoencoder to select the input sequence into the vector and predict the input again. Figure 3 is the model structure. The two algorithms can be exploited as pre-training algorithms for post-supervised sequence learning algorithms, enabling powerful performance in many classification tasks. However, applicability is limited severely due to the need for millions of intra-domain documents to achieve good performance. Howard and Ruder BIB007 proposed the Universal Language Model Fine-tuning (ULMFiT), which can be exploited to NLP. ULMFit recommends training language models on very large corpus to contain more semantic information and use it as the backbone of the classifier. When applying it to different FIGURE 2. Schematic diagram of the process about combining other tasks with the target task BIB005 . FIGURE 3. Self-encoder model structure  . datasets, you only need to fine-tune the parameters of the language model to take these differences into account, then add a classifier layer at the top of the language model and train only that layer. Howard J et al. have provided pretrained language models. Figure 4 is a training procedure for ULMFiT.