<jupyter_start><jupyter_text>The Amazon reviews polarity dataset is constructed by taking review score 1 and 2 as negative, and 4 and 5 as positive. Samples of score 3 is ignored. In the dataset, class 1 is the negative and class 2 is the positive. Each class has 1,800,000 training samples and 200,000 testing samples.The files train.csv and test.csv contain all the training samples as comma-sparated values. There are 3 columns in them, corresponding to class index (1 or 2), review title and review text. The review title and text are escaped using double quotes ("), and any internal double quote is escaped by 2 double quotes (""). New lines are escaped by a backslash followed with an "n" character, that is "\n". Load some awesome stuff<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline
from fastai import *
from fastai.text import *
from fastai.callbacks import *<jupyter_output><empty_output><jupyter_text>This block of code links your google drive to the your colab notebook<jupyter_code>import os
from google.colab import drive
drive.mount('/content/drive')
root_dir = Path('/content/drive/My Drive')
if not os.path.isdir(root_dir/'Amazon_Sentiment'):
  base_dir = root_dir/'Amazon_Sentiment2'
  base_dir.mkdir(parents=True, exist_ok=True)
else:
  base_dir = root_dir/'Amazon_Sentiment2' 
base_dir
# check if the dirctory exists if not make it
if not os.path.isdir(base_dir/'dataset'): (base_dir/'dataset').mkdir(exist_ok=True, parents=True)
data_dir = base_dir/'dataset'
data_dir<jupyter_output><empty_output><jupyter_text>Download the dataset from [Amazon reviews](https://s3.amazonaws.com/fast-ai-nlp/amazon_review_polarity_csv.tgz) into your google drive<jupyter_code>!wget https://s3.amazonaws.com/fast-ai-nlp/amazon_review_polarity_csv.tgz -P '/content/drive/My Drive/Amazon_Sentiment/dataset'
os.chdir(data_dir)
!tar -xf '/content/drive/My Drive/Amazon_Sentiment/dataset/amazon_review_polarity_csv.tgz'
data_dir.ls()
tmp_root = (data_dir/'amazon_review_polarity_csv')
tmp_root.ls()<jupyter_output><empty_output><jupyter_text>View what is inside the train and test dataset csv files<jupyter_code>def read_csv(path: Path, names :List[str]=['class', 'review_title', 'review']):
  df = pd.read_csv(path, header=None, names=names)
  # review_1 = df[['rating', 'review']]
  # review_2 = df[['rating', 'review_1']].rename(columns={'review_1':'review'})
  # return pd.concat([review_1, review_2])
  return df
train = read_csv(tmp_root/'train.csv')
train = train.loc[:1000000, :].copy()
train.head()
test = read_csv(tmp_root/'test.csv')
test.head()<jupyter_output><empty_output><jupyter_text>I noticed that the train and test sets has some missing values so i had to remove them. Neural nets do not like missing values in a dataset so the databunch creating will throw an error.<jupyter_code>train_no_na = train.loc[~train.review.isna()]
test_no_na = test.loc[~test.review.isna()]
np.random.seed(2)
bs = 16
train_no_na.iloc[0]
len(train_no_na)
len(test_no_na)<jupyter_output><empty_output><jupyter_text>The size of the databunch is atleast 2GB so i had to reduce the size of the training corpus by 30% so that it'll fit into the RAM and get serialized without crashing the session due to an out of memory(OOM) error.<jupyter_code>train_no_na_70_pct = train_no_na[:(int(0.7*len(train_no_na)))]
lm_data = (TextList.from_df(train_no_na_70_pct, cols=1)
                   .split_by_rand_pct(0.1)
                   .label_for_lm()
                   .databunch(bs=bs))<jupyter_output><empty_output><jupyter_text>Save the language model databunch os we won't have to go through this process again<jupyter_code># specify the path to save the data
if not os.path.isdir(base_dir/'data'):  (base_dir/'data').mkdir(parents=True, exist_ok=True); data_dir = (base_dir/'data') 
else: data_dir = base_dir/'data'
data_dir
lm_data.save(data_dir/'lm_data.pkl')
lm_data = load_data(data_dir, 'lm_data.pkl', bs=bs)
lm_data.show_batch()<jupyter_output><empty_output><jupyter_text>Language Model Training  Use the frozen pretrained weights to create a baseline language model then finetune the pretrained weights on the new dataset We'll be using tranfer learning using weights trained on the wiki-text dataset. We'll also be using dropout to decrease overfitting<jupyter_code>lm_learner = language_model_learner(lm_data, AWD_LSTM, drop_mult=0.3)
lm_learner.lr_find()
lm_learner.recorder.plot(suggestion=True)
lm_learner.recorder.plot(skip_end=15, suggestion=True)<jupyter_output>Min numerical gradient: 1.32E-06
Min loss divided by 10: 8.32E-03<jupyter_text>Training the network will update only the parameters of only the last few layers. These last few layers are that learn the features of the `data_lm` using the pretrained weights trained on wiki-text<jupyter_code>lm_learner.fit_one_cycle(1, 1e-2, moms=(0.8, 0.7))<jupyter_output><empty_output><jupyter_text>Export the language model so that it'll be saved in a `.pkl` format. I've found this can be a bit time consuming to load than using the `.save` method that saves it in a `.pth` format<jupyter_code># specify the path to save the models
if not os.path.isdir(base_dir/'models'):  (base_dir/'models').mkdir(parents=True, exist_ok=True); model_dir = base_dir/'models'  
else: model_dir = base_dir/'models'
model_dir
lm_learner.export(model_dir/'lm_head.pkl')<jupyter_output><empty_output><jupyter_text>Fine-Tune the language model load the learner. This just loads the weights that the learner learnt but it doesn't load the data being learnt with it<jupyter_code>lm_learner = load_learner(model_dir, 'lm_head.pkl')<jupyter_output><empty_output><jupyter_text>Load the data being learnt<jupyter_code>lm_learner.data = lm_data<jupyter_output><empty_output><jupyter_text>Unfreeze all the layers of the network so we can update all their parameters during training<jupyter_code>lm_learner.unfreeze()
lm_learner.lr_find()
lm_learner.recorder.plot(skip_end=15, suggestion=True)<jupyter_output><empty_output><jupyter_text>Use a callback to save the model after every epoch<jupyter_code># change the directory to where we want the callback to save the every epoch model
os.chdir(model_dir)
callbacks = [SaveModelCallback(lm_learner, every='epoch', monitor='valid_loss', name='finetuned_lm')]
lm_learner.fit_one_cycle(10, 1e-3, moms=(0.8, 0.7), callbacks= callbacks)<jupyter_output><empty_output><jupyter_text>Each epoch was taking to long so i cancelled it after the first. Thankfully we have callbacks that save the model after every epoch :). We can jsut load the model after the last saved epoch<jupyter_code>lm_learner.load('finetuned_lm_0')<jupyter_output><empty_output><jupyter_text>Language Model Prediction<jupyter_code>WORD = 'Book'
# how many next words do we want to predict
NUM_OF_NEW_WORDS = 20
# Specify the number of sentences that the model can make using the WORD as a starter
NUM_OF_SENTENCES = 2
print('\n'.join(lm_learner.predict(WORD, NUM_OF_NEW_WORDS, temperature = 0.6) for _ in range(NUM_OF_SENTENCES)))<jupyter_output>Book was ok but the book was just not well written . Also the book was mostly written by a
Book Sucks xxbos This book is a waste of time and money . It is not worth the<jupyter_text>We can see that the language model we have has learnt how to speak in 'Amazon review English' We can then export the encoder from the finetuned language model<jupyter_code>lm_learner.save_encoder('lm_fine_tuned_enc')<jupyter_output><empty_output><jupyter_text>Let's Build the Classifier<jupyter_code>len(lm_data.vocab.itos)<jupyter_output><empty_output><jupyter_text>We have to sort our testing and training dataframes<jupyter_code>train = pd.read_csv(tmp_root/'train.csv', header=None, names = ['pos/neg', 'title', 'review'])
train.head()
len(train)<jupyter_output><empty_output><jupyter_text>ensure to use the same vocab when building the IteList<jupyter_code>clas_data = (TextList.from_df(train_no_na, cols = [1], vocab=lm_data.vocab))
clas_data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0984553560215999, 'fraction_numerical': 0.012306919502699987, 'mean_word_length': 6.367252543940795, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '74805', 'license': 'MIT', 'n_tokens_mistral': 2768, 'n_tokens_neox': 2659, 'n_words': 780, 'path': 'Amazon_Sentiment.ipynb', 'repo_name': 'rajibmondal/nlp-tutorial', 'size': '7963'}