<jupyter_start><jupyter_text>Sentiment analysis of weather related tweetsThis notebook explores some initial modeling of sentiment analysis of weather related tweets. The datasets can be downloaded from Kaggle's "[Partly Sunny with a Chance of Hashtags](https://www.kaggle.com/c/crowdflower-weather-twitter/data)" competition.Since the competition is over we will restrict ourselves to look at the training set only in order to have a good basis for benchmarking. We will make heavy use of *nltk* as a natural language processing (NLP) library to build a *bag-of-words* model that we we will use to train a Multinomial Naive Bayes classifier. Retrieving and basic cleansing of the data<jupyter_code>import pandas as pd
pd.options.display.encoding = 'ascii'
### import the different corpora and packages from nltk as you need them
# import nltk
# nltk.download()
raw_df = pd.DataFrame.from_csv('./datasets/train.csv')
raw_df.head(5)<jupyter_output><empty_output><jupyter_text>For completeness we will also show a snippet from the test set to see what kind of features we are allowed to use.<jupyter_code>test_df = pd.DataFrame.from_csv('./datasets/test.csv')
test_df.head(2)<jupyter_output><empty_output><jupyter_text>Label creation for Sentiment categoryWe start out by trying to model the sentiment category of a tweet. To this end we start by replacing the maximum vlaue of the sentiment fields s1-s5 with the corresponding category<jupyter_code>def create_label(row):
    # row is of time pandas.Series need to cast to a list.
    lst = row.tolist()
    return lst.index(max(lst))+1
# apply defaults to work on columns rather than rows. Use axis = 1 to cahnge to row application.
label_df = raw_df[['s1','s2','s3','s4','s5']].apply(create_label, axis=1)
label_df.head(5)<jupyter_output><empty_output><jupyter_text>Tokenizing, stemming and term countingIn order to use the tweets as input for a machine learning algorithm we need to convert them into numerical features. One way to do this is to chop up the tweets and count the number of words within a tweet and turn them into a large sparse vector whose length is size of the vocabulary of all tweets.We start with a very simple way of counting the terms in the total of all tweets and count the number of terms.<jupyter_code>from collections import Counter

full_tweet_string = raw_df.tweet.apply(lambda t: t.lower() + " ").sum()
Counter(full_tweet_string.split()).most_common()[:25]<jupyter_output><empty_output><jupyter_text>The simple counter already indicates a problem. There are a lot of very common words that have obiously no signal, such as 'the', 'at', 'of', etc. We need to remove those stopwords. Another problem can be seem by comparing '@mention' and '@mention:' which should clearly be identified as the same word, meaning we need to remove the punctuation. Finally we might want to identify 'storm' and 'stormy' as the same and thus require stemming techniques.The following code snippet provides a tokenizer that does all of the above<jupyter_code>from nltk.corpus import stopwords
from nltk.stem.snowball import EnglishStemmer
from nltk import word_tokenize
from nltk.util import ngrams
import string

def custom_tokenizer(document_string):
    
    # define the stop word vocabulary
    stops = [unicode(word) for word in stopwords.words('english')] \
        + ["''", "``", 're:', 'fwd:', '-', '@mention', '@mention:', 'mention', 'link', ':', 'f.', '&'] 
    
    # create a default stemmer
    stemmer = EnglishStemmer()
    
    # return the stemmed list of words
    tokens = [stemmer.stem(unicode(word)) for word in word_tokenize(document_string.lower()) \
            if not (unicode(word.lower()) in stops or unicode(word.lower()) in list(string.punctuation))]
    
    return tokens + list(ngrams(tokens, 2))<jupyter_output><empty_output><jupyter_text>Let's have a look at the word counts again<jupyter_code>Counter(custom_tokenizer(full_tweet_string)).most_common()[:25]<jupyter_output><empty_output><jupyter_text>This looks already much better and we can identify weather realated terms such as 'storm', 'sunni', etc.This will be a good starting point for the rest of the model. Model generationOne step that is missing is how to use the *custom_tokenizer* to actually create feature vectors. Luckily enough sklearn provides use with the right functionality to do just that.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(min_df = 100, max_df = 10000, tokenizer = custom_tokenizer)<jupyter_output><empty_output><jupyter_text>To ensure we have always the same vectorization options we need to "fit" the vectorizer to the set so we can vectorize any new examples according to this. We do it here on the full set from which we will also extract the cross-validation data and we should be aware of this bias, as it necessarily eliminates the possibility of encountering previously unseen terms when we run our test set predicition.<jupyter_code># fit the vectorizer to the full set
X = vectorizer.fit_transform(raw_df.tweet.tolist())
X.shape<jupyter_output><empty_output><jupyter_text>Splitting the set and training the modelAs the training data and the labels are now spearate and of different type it is annoying to split the training and test-sets manually. Again sklearn has a tool for us that we can use: the cross_validation API.<jupyter_code>from sklearn import cross_validation

df_train, df_test, y_train, y_test = cross_validation.train_test_split(raw_df, label_df, test_size = 0.3)<jupyter_output><empty_output><jupyter_text>To avoid the above case of having the "perfect" vocabulary let us now train a new vectorizer using only the training set<jupyter_code>X_train = vectorizer.fit_transform(pd.DataFrame(df_train)[0].tolist())
X_test = vectorizer.transform(pd.DataFrame(df_test)[0].tolist())<jupyter_output><empty_output><jupyter_text>We can now start to train the model. As this is a highly sparse problem it lends itself to be tackled using a Naive Bayes classifier.<jupyter_code>from sklearn.naive_bayes import MultinomialNB

multi_nb_clf = MultinomialNB()
multi_nb_clf.fit(X_train, y_train)

multi_nb_clf.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>With only single terms (monograms), bigrams (interestingly enough they seem not to have any impact at this level) and no additionally fine-grained modeling we already classify 60% of the tweets into the right category. This is impressive in so far that there are a lot of easy and obvious ways to improve the classification. Some ways to improve are* include higher n-grams in the tokenizer* train models per state or per city* include state and city as features* include TF-IDF vectorizers* use a different classifier? MiscelleneaReferencesare mostly given throughout the text. But importantly- [1] [Partly Sunny with a Chance of Hashtags](https://www.kaggle.com/c/crowdflower-weather-twitter)- [2] [scikit-learn](http://scikit-learn.org/stable/index.html)- [3] [nltk](http://www.nltk.org/index.html)Stylesheet<jupyter_code>from IPython.core.display import HTML

def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08009406556923503, 'fraction_numerical': 0.004841610181214553, 'mean_word_length': 5.449598572702944, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 2, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '381423', 'license': 'CC0-1.0', 'n_tokens_mistral': 2167, 'n_tokens_neox': 2078, 'n_words': 844, 'path': 'Data-Analytics/Sentiment analysis of weather related tweets.ipynb', 'repo_name': 'jotterbach/Data-Exploration-and-Numerical-Experimentation', 'size': '7229'}