This paper describes a simple and principled approach to automatically construct sentiment lexicons using distant supervision . We induce the sentiment association scores for the lexicon items from a model trained on a weakly supervised corpora . Our empirical findings show that features extracted from such a machine-learned lexicon outperform models using manual or other automatically constructed sentiment lexicons . Finally , our system achieves the state-of-the-art in Twitter Sentiment Analysis tasks from Semeval-2013 and ranks 2nd best in Semeval-2014 according to the average rank . One of the early and rather successful models for sentiment analysis ( Pang and Lee , 2004 ; Pang and Lee , 2008 ) relied on manually constructed lexicons that map words to their sentiment , e.g. , positive , negative or neutral . The document-level polarity is then assigned by performing some form of averaging , e.g. , majority voting , of individual word polarities found in the document . These systems show an acceptable level of accuracy , they are easy to build and are highly computationally efficient as the only operation required to assign a polarity label are the word lookups and averaging . However , the information about word polarities in a document are best exploited when using machine learning models to train a sentiment classifier . In fact , most successful sentiment classification systems rely on supervised learning . Interestingly , a simple bag of words model using just unigrams and bigrams with an SVM has shown excellent results ( Wang and Manning , 2012 ) performing on par or beating more complicated models , e.g. , using neural networks ( Socher et al . , 2011 ) . Regarding Twitter sentiment analysis , the top performing system ( Mohammad et al . , 2013 ) from Semeval-2013 Twittter Sentiment Analysis task ( Nakov et al . , 2013 ) follows this recipe by training an SVM on various surface form , sentiment and semantic features . Perhaps , the most valuable finding is that sentiment lexicons appear to be the most useful source of features accounting for over 8 point gains in the F-measure on top of the standard feature sets . Sentiment lexicons are mappings from words to scores capturing the degree of the sentiment expressed by a given word . While several manually constructed lexicons are made available , e.g. , the MPQA ( Wilson et al . , 2005 ) , the Bing and Liu ( Hu and Liu , 2004 ) and NRC Emoticon ( Mohammad and Turney , 2013 ) lexicons , providing high quality word-sentiment associations compiled by humans , still their main drawback is low recall . For example , the largest NRC Emoticon lexicon contains only 14k items , whereas tweets with extremely sparse surface forms are known to form very large vocabularies . Hence , using larger lexicons with better recall has the potential of learning more accurate models . Extracting such lexicons automatically is a challenging and interesting problem ( Lau et al . , 2011 ; Bro and Ehrig , 2013 ; Liu et al . , 2013 ; Tai and Kao , 2013 ; Yang et al . , 2014 ; Huang et al . , 2014 ) . However , different from previous work our goal is not to extract human-interpretable lexicons but to use them as a source of features to improve the classifier accuracy . Following this idea , the authors in ( Mohammad et al . , 2013 ) use features derived from the lexicons to build a state-of-the-art sentiment classifier for Twitter . They construct automatic lexicons using noisy labels automatically inferred from emoticons and hashtags present in the tweets . The wordsentiment association scores are estimated using pointwise mutual information ( PMI ) computed between a word and a tweet label . While the idea to model statistical correlations between the words and tweet labels using PMI or any other metric is rather intuitive , we believe there is a more effective way to exploit noisy labels for estimating the word-sentiment association scores . Our method relies on the idea of distant supervision ( Marchetti-Bowick and Chambers , 2012 ) . We use a large distantly supervised Twitter corpus , which contains noisy opinion labels ( positive or negative ) to learn a supervised polarity classifier . We encode tweets using words and multi-word expressions as features ( which are also entries in our lexicon ) . The weights from the learned model are then used to define which lexicon items to keep , i.e. , items that constitute a good sentiment lexicon . The scores for the lexicon items can be then directly used to encode new tweets or used to derive more advanced features . Using machine learning to induce the scores for the lexicon items has an advantage of learning the scores that are directly optimized for the classification task , where lexicon items with higher discriminative power tend to receive higher weights . To assess the effectiveness of our approach , we reimplemented the state-of-the-art system ranking 1st in Semeval-2013 Twitter Sentiment Analysis challenge and used it as our baseline . We show that adding features from our machine-learned sentiment lexicon yields better results than any of the automatic PMI lexicons used in the baseline and all of them combined together . Our system obtains new state-of-the-art results on the SemEval-2013 message level task with an F-score of 71.32 -a 2 % of absolute improvement over the previous best system in SemEval-2013 . We also evaluate the utility of the ML lexicon on the five test sets from a recent Semeval-2014 task showing significant improvement over a strong baseline . Finally , our system shows high accuracy among the 42 systems participating in the Semeval-2014 challenge ranking 2nd best according to the average rank across all test sets . We demonstrated a simple and principled approach grounded in machine learning to construct sentiment lexicons . We show that using off-the-shelf machine learning tools to automatically extract lexicons greatly outperforms other automatically constructed lexicons that use pointwise mutual information to estimate sentiment scores for the lexicon items . We have shown that combining our machinelearned lexicon with the previous best system yields state-of-the-art results in Semeval-2013 gaining over 2 points in F-score and ranking our system 2nd according to the average rank over the five test sets of Semeval-2014 . Finally , our ML-based lexicon shows excellent results when added on top of the current state-of-the-art NRC system . While our experimental study is focused on Twitter , our method is general enough to be applied to sentiment classification tasks on other domains . In the future , we plan to experiment with constructing ML lexicons from larger Twitter corpora also using hashtags . Recently , deep convolutional neural networks for sentence modelling ( Kalchbrenner et al . , 2014 ; Kim , 2014 ) have shown promising results on several NLP tasks . In particular , ( Tang et al . , 2014 ) showed that learning sentiment-specific word embeddings and using them as features can boost the accuracy of existing sentiment classifiers . In the future work we plan to explore such approaches .
While sentiment lexicons are useful for building accurate sentiment classification systems, existing methods suffer from low recall or interpretability.
They propose to use Twitter's noisy opinion labels as distant supervision to learn a supervised polarity classifier and use it to obtain sentiment lexicons.
Using the obtained lexicon with an existing model achieves the state-of-the-art on the SemEval-13 message level task and outperforms baseline models in several other datasets.