<jupyter_start><jupyter_text>**Fellowship.ai Challenge** * **Name**: <NAME>* **Email** : <EMAIL>* **Region**: ASIA - IST  Topic Chosen - ULMFiT Sentiment : Apply a supervised or semi-supervised ULMFiT model to Twitter US Airlines Sentiment * Environment Used for notebook - Kaggle Kernels* Dataset Used : twitter-airline-sentiment on kaggle (https://www.kaggle.com/crowdflower/twitter-airline-sentimentTweets.csv)* Method used for sentiment analysis: ULMFit method (making use of RNNs and LSTMs)<jupyter_code># This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the read-only "../input/" directory
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using "Save & Run All" 
# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session<jupyter_output>/kaggle/input/twitter-airline-sentiment/database.sqlite
/kaggle/input/twitter-airline-sentiment/Tweets.csv<jupyter_text>**Importing dependencies**<jupyter_code>#utilities for data processing and algebra
import numpy as np 
import pandas as pd 

#for specialized container datatypes
import collections

#for plotting and data visualization
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

#for feature preprocessing
import re

#sklearn for machine learning
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.utils.multiclass import unique_labels

#extension to reload modules before executing user code
%reload_ext autoreload
%autoreload 2

#importing os to analyse and organize the directory structure
import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))<jupyter_output>/kaggle/input/twitter-airline-sentiment/database.sqlite
/kaggle/input/twitter-airline-sentiment/Tweets.csv<jupyter_text>**Importing fastai for Natural Language Processing and ULMFiT**<jupyter_code>from fastai import *
from fastai.text import *<jupyter_output><empty_output><jupyter_text>**Importing utilities for collecting/checking fastai user environment**<jupyter_code>import fastai.utils.collect_env
fastai.utils.collect_env.show_install()<jupyter_output>```text
=== Software === 
python        : 3.7.6
fastai        : 1.0.61
fastprogress  : 0.2.3
torch         : 1.5.1
nvidia driver : 418.67
torch cuda    : 10.1 / is available
torch cudnn   : 7603 / is enabled

=== Hardware === 
nvidia gpus   : 1
torch devices : 1
  - gpu0      : 16280MB | Tesla P100-PCIE-16GB

=== Environment === 
platform      : Linux-4.19.112+-x86_64-with-debian-buster-sid
distro        : #1 SMP Sat Jun 13 11:04:33 PDT 2020
conda env     : Unknown
python        : /opt/conda/bin/python3.7
sys.path      : /kaggle/working
/kaggle/lib/kagglegym
/kaggle/lib
/opt/conda/lib/python37.zip
/opt/conda/lib/python3.7
/opt/conda/lib/python3.7/lib-dynload

/root/.local/lib/python3.7/site-packages
/opt/conda/lib/python3.7/site-packages
/src/bq-helper
/opt/conda/lib/python3.7/site-packages/IPython/extensions
/root/.ipython
```

Please make sure to include opening/closing ``` when you paste into forums/github to make the reports appear formatted as code sections.

Optional package(s)[...]<jupyter_text>**Data Representation and pre-processing**<jupyter_code>path = Path('../input/twitter-airline-sentiment/')
file_name = 'Tweets.csv'
path.ls()<jupyter_output><empty_output><jupyter_text>**Shuffling & reading the data into a pandas dataframe**<jupyter_code>file_path = path / file_name
df_tweets = pd.read_csv(file_path)
df_tweets = df_tweets.reindex(np.random.permutation(df_tweets.index)) 
df_tweets.head()<jupyter_output><empty_output><jupyter_text>**analysing features and plotting relevant ones**<jupyter_code>df_tweets.info()
df_tweets.describe()<jupyter_output><empty_output><jupyter_text>We see that the two most relevant features for us to perform sentiment analysis would be the columns 'airline_sentiment' and the 'text' that is contained in the tweet. **Plotting the different sentiments wrt the types and the airline towards which these sentiments were directed.**<jupyter_code>df_tweets['airline_sentiment'].value_counts().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>From the above graph, we can clearly see that most of the tweets in the given dataset are negative, followed by neutral. The number of positive tweets are relatively fewer in number.<jupyter_code>df_tweets['airline'].value_counts().plot(kind='bar')
df_tweets.groupby(['airline', 'airline_sentiment']).size().unstack().plot(kind='bar', stacked=True)<jupyter_output><empty_output><jupyter_text>**From the above two graphs we can see that there is a clear correlation between the degree of passenger satisfaction and the specific airline they were flying with, hence while feature engineering, we would have to filter out the airline name from our dataset so that our model doesn't learn it as a parameter and make it a basis for prediction.** **Analysing the sentiment on the basis of the length of the tweet**<jupyter_code>df_tweets['tweet_length'] = df_tweets['text'].apply(len)
df_tweets.groupby(['tweet_length', 'airline_sentiment']).size().unstack().plot(kind='line', stacked=False)<jupyter_output><empty_output><jupyter_text>We see that there isn't alot of correation between the tweet length and the sentiment behind the tweet. The only conclusion that can safely be drawn from the above graph is the fact that the distribution is heavily skewed in the case of negative tweets, as the more dissatisfied the passenger, the more they have to say.  Lets check if any such conclusion can be drawn when it comes to average and median sentiment confidence as well.<jupyter_code>df_tweets[['tweet_length', 'airline_sentiment', 'airline_sentiment_confidence']].groupby(['tweet_length', 'airline_sentiment']).mean().unstack().plot(kind='line', stacked=False)
plt.title('Average Airline Sentiment vs tweet length')
df_tweets[['tweet_length', 'airline_sentiment', 'airline_sentiment_confidence']].groupby(['tweet_length', 'airline_sentiment']).median().unstack().plot(kind='line', stacked=False)
plt.title('Median Airline Sentiment vs tweet length')<jupyter_output><empty_output><jupyter_text>As no distinguishable and concrete relationship can be inferred from the confidence and tweet length, we will form a dataframe considering the text and airline_sentiment attribute.<jupyter_code>features = ['airline_sentiment','text']
df_needed = df_tweets[features]
pd.set_option('display.max_colwidth', 0)
df_needed.head()
df_needed.info()
df_needed.describe()<jupyter_output><empty_output><jupyter_text>**Checking for missing values in our data**<jupyter_code>df_needed.isna().sum()
#df_needed['airline_sentiment'].count()
df_needed['airline_sentiment'].value_counts()
sns.set(style = "whitegrid")
sns.set(rc={'figure.figsize': (12,10)})
sns.countplot(x="airline", hue = 'airline_sentiment', data=df_tweets)
plt.title("Airline Sentiment for each US Airline")<jupyter_output><empty_output><jupyter_text>As we see that customer sentiments are somewhat specific to the airline they are using to travel, hence we will have to remove the names of the airlines so that our predictive model is not influenced by the names of the airlines because of the high correlation.<jupyter_code>reg = r"@(VirginAmerica|SouthwestAir|united|AmericanAir|Delta|USAirways)"<jupyter_output><empty_output><jupyter_text>Writing a function to replace the airline name<jupyter_code>def text_filter(text):
    return re.sub(reg,'@airline',text, flags = re.IGNORECASE)
df_needed['text'] = df_needed['text'].apply(text_filter)
df_needed.head(15)<jupyter_output><empty_output><jupyter_text>We have successfully removed the airlines names from our dataframe.Now we convert the obtained dataframe into a databunch, as required by our training method.  ULMFiT training method **Splitting the dataset into train and test sets, initializing hyperparameters and converting it into a Databunch**<jupyter_code>train, test = train_test_split(df_needed, test_size =0.1)
moms = (0.8,0.7)
wd = 0.1<jupyter_output><empty_output><jupyter_text>*Tokenization*We read in the data and add new words to our dictionary, as well as create a representation of words using numbers.<jupyter_code>data = TextLMDataBunch.from_df(path, train_df = train, valid_df = test)
data.show_batch()<jupyter_output><empty_output><jupyter_text>**As we can see from the above dataframe, the following changes have been implemented for the ease of data modeling and model training:*** splitting on spaces* splitting on punctuation marks* several special tokens have been usd to replace the unknown tokens.* Contractions like "wouldn't", have been separated. Representation of unknown tokens can further be understood by looking at the below list for the entries beginning with symbols ('xx').<jupyter_code>data.vocab.itos[:10]<jupyter_output><empty_output><jupyter_text>Text representation post pre-processing<jupyter_code>print(data.train_ds[0][0])
print(data.train_ds[2][0])
print(data.train_ds[4][0])<jupyter_output>xxbos @airline our hold up now is the captain that was xxmaj late xxmaj flight ... his bag does n't fit .
xxbos @airline " xxmaj you down with xxup xxunk ? " " xxmaj yeah you know me ! "
xxbos xxmaj need more info on that ! @airline : @momsgoodeats xxmaj we have great xxunk xxunk . xxup pj 's not included . xxmaj we status match xxup aa . xxup ^kp ”<jupyter_text>Numerical representation post pre-processing<jupyter_code>print(data.train_ds[0][0].data[:10])
print(data.train_ds[2][0].data[:10])
print(data.train_ds[4][0].data[:10])<jupyter_output>[   2   10   75   84   73   58   26   13 1018   35]
[  2  10  62   5  17 313  39   6   0  15]
[  2   5  95 120 289  22  35  14  10  33]<jupyter_text>Language Model<jupyter_code>#learn = language_model_learner(data, AWD_LSTM, drop_mult=0.3)<jupyter_output><empty_output><jupyter_text>Defining the language model and setting the leaning rates.<jupyter_code>learn = language_model_learner(data, AWD_LSTM, drop_mult=0.5, model_dir='/tmp/models')
learn.freeze()
learn.lr_find()
learn.recorder.plot()
learn.recorder.plot(skip_end=15)<jupyter_output><empty_output><jupyter_text>Now, we fit the model for a few cycles by running 1 epoch and then unfreezing and running subsequent epochs to fine tune the same.<jupyter_code>moms = (0.8,0.7)
wd = 0.1
lr = 1.0E-02
learn.fit_one_cycle(1, lr, moms=moms, wd=wd)
learn.unfreeze()
learn.fit_one_cycle(10,lr,moms = moms, wd=wd)<jupyter_output><empty_output><jupyter_text>After training the model for 10 epochs, we see that the accuracy of our model is very low, lets check the prediction on a custom input.<jupyter_code>learn.predict("This airline is so bad! Never flying with them again!", n_words = 40)<jupyter_output><empty_output><jupyter_text>**The language model doesn't perform too well due to a variety of reasons -*** As the tweets normally have no fixed format and include informal, often incorrect gramatical structures.* Words in tweets often don't follow correct spellings and generic language conventions.* As seen above, figures of speech like 'sarcasm' are difficult to account for.<jupyter_code>learn.save_encoder('fine_tuned_enc')<jupyter_output><empty_output><jupyter_text>**Fine tuning our classifier**Now we will repeat the process of creating and training a databrunch. Then train our LSTM language model all over again, but this time to predict the sentiments of the tweet and not just the word/words that follow.This model architecture offers alot of advantage over its traditional language counterpart which uses the conventional bag of words representation for predictions. **Splitting into test, train and validation sets**<jupyter_code>train_valid, test = train_test_split(df_needed, test_size = 0.1)
train, valid = train_test_split(train_valid, test_size = 0.1) 
data_classifier = TextClasDataBunch.from_df(path, train_df = train, valid_df = valid, test_df = test, vocab = data.train_ds.vocab, text_cols = 'text', label_cols = 'airline_sentiment', bs = 24)
data_classifier.show_batch()
learn = text_classifier_learner(data_classifier, AWD_LSTM, drop_mult = 0.5, model_dir = '/tmp/models')
learn.load_encoder('fine_tuned_enc')
learn.freeze()
learn.lr_find()
learn.recorder.plot()
learn.recorder.plot(skip_end = 15)<jupyter_output><empty_output><jupyter_text>**We train by gradually unfreezing layers and training our model one epoch at a time, in accordance to the suggestions in the fast.ai ULMFiT paper.**<jupyter_code>lr = 1.0E-03
learn.fit_one_cycle(1,lr,moms=moms, wd=wd)
learn.save('first')
#learn.load('first')
learn.freeze_to(-2)
lr/=2
learn.fit_one_cycle(1,slice(lr/(2.6*4), lr), moms=moms, wd=wd)
learn.save('second')
#learn.load('second')
learn.freeze_to(-3)
lr /=2
learn.fit_one_cycle(1,slice(lr/(2.6*4), lr), moms=moms, wd=wd)
learn.unfreeze()
lr /=5
learn.fit_one_cycle(3, slice(lr/(2.6**4),lr), moms=moms, wd=wd)<jupyter_output><empty_output><jupyter_text>Our model has now been trained, lets try to get a prediction for the same custom input we gave earlier.<jupyter_code>learn.predict("This airline is so bad! Never flying with them again!")<jupyter_output><empty_output><jupyter_text>We see that this model performs way better than our conventional language model. And hence we can now evaluate it's performance on the test set.  Evaluate performance on test set.<jupyter_code>print(test)
test['airline_sentiment'].value_counts().plot(kind='bar')
test['airline_sentiment'].value_counts()<jupyter_output><empty_output><jupyter_text>**Predicting the sentiments of tweets in the test set**<jupyter_code>test['predicted']=test['text'].apply(lambda row: str(learn.predict(row)[0]))
print(test)
test.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1464 entries, 2437 to 5177
Data columns (total 3 columns):
 #   Column             Non-Null Count  Dtype 
---  ------             --------------  ----- 
 0   airline_sentiment  1464 non-null   object
 1   text               1464 non-null   object
 2   predicted          1464 non-null   object
dtypes: object(3)
memory usage: 45.8+ KB<jupyter_text>**Evaluation**Evaluation metric used - AccuracyAs the predicted sentiments would be either positive, negative or neutral, we can easily compare the predictions to the labels provided in the test set and check the number of times our model makes the correct prediction.<jupyter_code>print("Test Accuracy: ", accuracy_score(test['airline_sentiment'], test['predicted']))<jupyter_output>Test Accuracy:  0.8224043715846995<jupyter_text>**We see that our model gives us an accuracy of over 80% on the test set, which is quite good, considering the kind of data set and its anomalies.**<jupyter_code>test.head(25)<jupyter_output><empty_output><jupyter_text>**Now we plot the confusion matrix to see what the areas of misclassifiacation are and the reason for their existence.**<jupyter_code>def plot_matrix(y_true, y_pred, classes, normalize = False, title = None, cmap=plt.cm.Blues):
    """
    This function plots the confusion matrix.
    Normalization can be applied by setting the Normalize parameter
    """
    
    if not title:
        if normalize:
            title = "Normalized Confusion Matrix"
        else:
            title='Not Normalized Confusion Matrix'
            
    confusion = confusion_matrix(y_true, y_pred)
    
    
    fig,ax = plt.subplots()
    im = ax.imshow(confusion, interpolation='nearest', cmap=cmap)
    ax.figure.colorbar(im, ax=ax)
    
    #display all ticks
    ax.set(xticks = np.arange(confusion.shape[1]),
          yticks = np.arange(confusion.shape[0]),
          xticklabels = classes, yticklabels = classes,
          title = title,
          ylabel = 'True Label',
          xlabel = 'Predicted Label')
    
    # display the count annotations by looping
    dtype = '.2f' if normalize else 'd'
    thresh = confusion.max() / 2.
    for i in range(confusion.shape[0]):
        for j in range(confusion.shape[1]):
            ax.text(j,i, format(confusion[i,j],dtype),
                   ha = 'center',va = 'center',
                   color = 'white' if confusion[i,j]> thresh else 'black')
    
    
    fig.tight_layout()
    return ax
plot_matrix(test['airline_sentiment'], test['predicted'], classes=['negative','neutral','positive'], title = 'Airline Tweets Sentiment Analysis Confusion Matrix')
plt.show()<jupyter_output><empty_output><jupyter_text>**True negatives - ** Have a very high prediction accuracy.The model has a tough time classifying neutral tweets, often branding them as negative despite having an overall accuracy of over 80%.But in our dataset, some of the tweets carry negative sentiment aimed at other twitter users instead of the airline. In addition, there are other things to consider with informal writing such as sarcasm, and improper gramatical structures that might en up confusing our model, leading to inaccurate predictions.Moreover some of the negative tweets have been mislabelled in the dataset as neutral and vice versa, which can be another factor hampering the accuracy of our model. The overall accuracy is good for true positives, but a vast majority of positive tweets get classified as negative. Lets look into them in more detail.<jupyter_code>test.loc[(test['airline_sentiment']=='positive') & (test['predicted']=='negative')]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10091590341382181, 'fraction_numerical': 0.016153205661948375, 'mean_word_length': 5.354850088183421, 'pattern_counts': {'":': 0, '<': 133, '<?xml version=': 0, '>': 134, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '34', 'dir': 'jupyter-structured-clean-dedup', 'id': '487438', 'license': 'MIT', 'n_tokens_mistral': 5907, 'n_tokens_neox': 5554, 'n_words': 1764, 'path': 'Sentiment Analysis of Airline Tweets/US Airlines Sentiment Analysis (ULMFIT Method).ipynb', 'repo_name': 'gauraverma33/Data-is-Beautiful', 'size': '18031'}