<jupyter_start><jupyter_text>Text Classification using ULMFiTHere we are going to prepare the classification model for classifying text as etiher **negative**, **positive** or **neutral** with the help of transfer learning. Since the text we are dealing with is in English we can use an already prepared language model to combine with our task. The structure of the text, language and semantic rules are same so it would save a lot of time and compute resources by using transfer learning rather than training model from scratch.ULMFiT stands for Universal Language Model Fine-tuning and it is introduced by the `fastai` library.It is well suited for tasks like text classification. Some of the benefits of using transfer learning are:1. We don't have to train the model from scratch, so it saves time.2. Similarly it saves compute resources.3. It is a boon for small datasets. 4. The accuracy obtaines is good.We are not going to go in details of the ULMFiT model here but [here](http://nlp.fast.ai/) is amazing documentation and the [paper](https://arxiv.org/abs/1801.06146).This the basic flow we are going to follow:With this approach no domain, label or **preprocessing** is required. So we don't need to remove stopwords and punctuations(although it is a good practise to do so) becuase that might lead to loss of information or pattern. ULMFiT will handle semantics on it's own. Uploading the dataset<jupyter_code># Upload your Kaggle API in order to download the required dataset.
from google.colab import files
files.upload()  #this will prompt you to upload the kaggle.json
# Preparing the API for downloading
!mkdir -p ~/.kaggle
!cp kaggle.json ~/.kaggle/
!ls ~/.kaggle
!chmod 600 /root/.kaggle/kaggle.json  # set permission
!kaggle datasets download -d crowdflower/twitter-airline-sentiment
!unzip /content/twitter-airline-sentiment.zip
# importing the required libraries

import pandas as pd
import fastai
from fastai import *
from fastai.text import *
data=pd.read_csv('/content/Tweets.csv')<jupyter_output><empty_output><jupyter_text>Although the distribution of labels here is unbalanced but it will be handled with transfer learning and ULMFiT. If were using a simpler classification model we would need to balance the data first.<jupyter_code>#checking the distribution of labels
data.shape
data.airline_sentiment.value_counts()<jupyter_output><empty_output><jupyter_text>Picking only the `text` and `airline sentiment` from the data.<jupyter_code>data=data[['airline_sentiment','text']]
data.head(10)<jupyter_output><empty_output><jupyter_text>Splitting the data into training and validation sets<jupyter_code>from sklearn.model_selection import train_test_split

#dropping null values
data.dropna(inplace=True)

# split data into training and validation set
df_trn, df_val = train_test_split(data, stratify = data['airline_sentiment'], test_size = 0.2, random_state = 12)
#checking the shape of training and validation set

df_trn.shape, df_val.shape<jupyter_output><empty_output><jupyter_text>Data for Language Model`TextLMBunch` will apply some preprocessing tasks. Thus we don't need to do preprocessing here.<jupyter_code># Language model data
data_lm = TextLMDataBunch.from_df(train_df = df_trn, valid_df = df_val, path = "")<jupyter_output><empty_output><jupyter_text>This is how the tokens are created. The `xx...` here hold special meaning and pattern which will help the network to learn.More information [here](https://docs.fast.ai/text.transform.html).<jupyter_code>data_lm.train_ds.vocab.itos[:10]
# the idx is the position relative to which the text is stored
data_lm.show_batch()<jupyter_output>/usr/local/lib/python3.6/dist-packages/fastai/text/data.py:339: UserWarning: This overload of nonzero is deprecated:
	nonzero()
Consider using one of the following signatures instead:
	nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:766.)
  idx_min = (t != self.pad_idx).nonzero().min()<jupyter_text>Creating the Language ModelUsing `pretrained=True` will let us load the pretrained model with with we don't need to train the model from scratch.We are downloading the weights from the pretrained model.<jupyter_code># create LM
lang_mod = language_model_learner(data_lm,  arch = AWD_LSTM, pretrained = True, drop_mult=1.)<jupyter_output>Downloading https://s3.amazonaws.com/fast-ai-modelzoo/wt103-fwd.tgz<jupyter_text>Fine Tuning Language ModelThe step-1 of utilizing the ULMFiT is to fine tune the langauge model.Here `lr_find` helps to find the optimal learning rate. The plot for the same is outputted by `lang_mod.recorder.plot(suggestion=True)`.We will follow these steps:- We are going to find the best learning rate (LR)- Adjust the last layer with the `fit_one_cycle`- Unfreeze all the layers- Find the new learning rate.<jupyter_code>lang_mod.lr_find()
lang_mod.recorder.plot(suggestion=True)<jupyter_output><empty_output><jupyter_text>Then we will train our model for 5 epochs, using the `fit_one_cycle` function. With this we will train, by now, only the last layer of our language model.<jupyter_code>lang_mod.fit_one_cycle(5, max_lr= 5e-02)<jupyter_output><empty_output><jupyter_text>We can continue to train the language model by gradually unfreezing some layers at a time. But the learning rate with each layer should be altered accordingly.<jupyter_code>#unfreezing one layer
lang_mod.freeze_to(-1)<jupyter_output><empty_output><jupyter_text>Finding optimal learning rate again.<jupyter_code>lang_mod.lr_find()
lang_mod.recorder.plot(skip_start=40, suggestion=True)
# train the language model
lang_mod.fit_one_cycle(3, slice(3e-3/(2.6**4), 1e-3))<jupyter_output><empty_output><jupyter_text>We can see that the train and valid loss are lower and less overfitted with high accuracy.  Same thing by unfreezing one more layer<jupyter_code>lang_mod.freeze_to(-2)
lang_mod.lr_find()
lang_mod.recorder.plot(skip_start=40, suggestion=True)
lang_mod.fit_one_cycle(3, slice(3e-3/(2.6**4), 1e-3))<jupyter_output><empty_output><jupyter_text>Last unfreeze<jupyter_code>lang_mod.unfreeze()
lang_mod.lr_find()
lang_mod.recorder.plot(skip_start=40, suggestion=True)
lang_mod.fit_one_cycle(3, slice(3e-3/(2.6**4), 1e-3))
#saving the langauge model
lang_mod.save_encoder('Encoder')<jupyter_output><empty_output><jupyter_text>The language model is ready and this will be used for the classification. Let's move onto step-2 of our task that is **Classification**.  Classification Model<jupyter_code># Classifier model data
data_clas = TextClasDataBunch.from_df(path = "", train_df = df_trn,  valid_df = df_val, vocab=data_lm.train_ds.vocab, bs=32)
data_clas.show_batch()<jupyter_output><empty_output><jupyter_text>Creating the classifier by passing the classifier model data.<jupyter_code>learn_classifier = text_classifier_learner(data_clas, drop_mult=0.7, arch = AWD_LSTM)<jupyter_output><empty_output><jupyter_text>Loading the saved langauge model.<jupyter_code>learn_classifier.load_encoder('Encoder')<jupyter_output><empty_output><jupyter_text>Similar to what we did with the language model, we are going to find the best learning rate (LR), adjust the last layer with the `fit_one_cycle`, unfreeze all the layers and find the new learning rate.<jupyter_code>learn_classifier.lr_find()
learn_classifier.recorder.plot(suggestion=True)
learn_classifier.fit_one_cycle(5, max_lr=1e-2, moms=(0.8,0.7))
learn_classifier.recorder.plot_losses()
#Unfreezing another layer
lang_mod.freeze_to(-1)

learn_classifier.lr_find()
learn_classifier.recorder.plot(suggestion=True)<jupyter_output><empty_output><jupyter_text>Let's test the classifier with some text. For that we are going to use the `predict` function.- Category 0: Negative- Category 1: Neutral- Category 2: Positive<jupyter_code>learn_classifier.predict('I loved the service. Cant wait to travel with you guys again :)')
learn_classifier.predict('Worst service ever. I am going to sue you')<jupyter_output><empty_output><jupyter_text>Storing the prediction for validation set.<jupyter_code>valid_preds, valid_label=learn_classifier.get_preds(ds_type=DatasetType.Valid, ordered=True)
valid_preds.shape<jupyter_output><empty_output><jupyter_text>Ratio per predicted class.<jupyter_code>preds=valid_preds.numpy()
print(np.mean(preds[:,0]))
print(np.mean(preds[:,1]))
print(np.mean(preds[:,2]))<jupyter_output>0.5958492
0.20865269
0.19549811<jupyter_text>Creating the dataframe<jupyter_code>df_val_pred=pd.DataFrame({'text':df_val.text, 'target':df_val.airline_sentiment, 'pred_target':valid_label})<jupyter_output><empty_output><jupyter_text>We will create a function `label_pred` to map numerical predictions to categories.<jupyter_code>def label_pred (row):
   if row['pred_target'] == 0 :
      return 'negative'
   if row['pred_target'] == 1:
      return 'neutral'
   else:
      return 'positive'
  
df_val_pred['pred_target']=df_val_pred.apply(lambda row: label_pred(row), axis=1)
df_val_pred.head()<jupyter_output><empty_output><jupyter_text>The distribution of predicted sentiments.<jupyter_code>df_val_pred.pred_target.value_counts().plot.bar()
from sklearn.metrics import confusion_matrix, accuracy_score

confusion_matrix(df_val.airline_sentiment,df_val_pred.pred_target)
accuracy_score(df_val.airline_sentiment,df_val_pred.pred_target)<jupyter_output><empty_output><jupyter_text>The accuracy score of 1.0 looks a bit suspicious but after testing the classifier with unseen data it seems like the classifier is able to classify the text well and transfer learning with ULMFiT has helped acheiving this accuracy. Before completing the notebook let's classify some more unseen text.<jupyter_code>learn_classifier.predict('United is thw worst airline. I want to urge all the passengers to never book your flights with them')
learn_classifier.predict('Virgin America is an airline I can trust anyday. I have been travelling with them for 5 years now.')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08923169993950393, 'fraction_numerical': 0.012704174228675136, 'mean_word_length': 6.665378670788254, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 94, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '431410', 'license': 'MIT', 'n_tokens_mistral': 3253, 'n_tokens_neox': 3130, 'n_words': 965, 'path': 'Model.ipynb', 'repo_name': 'yashika51/Twitter-US-Airlines-Sentiment', 'size': '9918'}