<jupyter_start><jupyter_text>Text Summarization of Amazon reviewsIn this notebook I will write summaries with the help of my Seq2Seq model in Summarizer.py.The model works impressively well in the end!<jupyter_code>import os

import pandas as pd
import numpy as np
import tensorflow as tf
import tensorflow_hub as hub
from collections import Counter

import Summarizer
import summarizer_data_utils
import summarizer_model_utils
print(tf.__version__)<jupyter_output>1.8.0<jupyter_text>The dataThe data we will be using with is a dataset from Kaggle, the Amazon Fine Food Reviews dataset.  It contains, as the name suggests, 570.000 reviews of fine foods from Amazon and summaries of those reviews. Our aim is to input a review (Text column) and automatically create a summary (Summary colum) for it.https://www.kaggle.com/snap/amazon-fine-food-reviews/data  Reading and exploring<jupyter_code># load csv file using pandas.
file_path = './Reviews.csv'
data = pd.read_csv(file_path)
data.shape
# we will only use the last two columns Summary (target) and Text (input).
data.head()
# check for missings --> got some in summary drop those. 
# 26 are missing, so we will drop those!
data.isnull().sum()
# drop row, if values in Summary is missing. 
data.dropna(subset=['Summary'],inplace = True)
# only summary and text are useful for us.
data = data[['Summary', 'Text']]
data.head()
# we will not use all of them, only short ones and ones of similar size. 
# choosing the ones that are of similar length makes it easier for the model to learn.
raw_texts = []
raw_summaries = []

for text, summary in zip(data.Text, data.Summary):
    if 100< len(text) < 150:
        raw_texts.append(text)
        raw_summaries.append(summary)
len(raw_texts), len(raw_summaries)
for t, s in zip(raw_texts[:5], raw_summaries[:5]):
    print('Text:\n', t)
    print('Summary:\n', s, '\n\n')<jupyter_output>Text:
 Great taffy at a great price.  There was a wide assortment of yummy taffy.  Delivery was very quick.  If your a taffy lover, this is a deal.
Summary:
 Great taffy 


Text:
 This taffy is so good.  It is very soft and chewy.  The flavors are amazing.  I would definitely recommend you buying it.  Very satisfying!!
Summary:
 Wonderful, tasty taffy 


Text:
 Right now I'm mostly just sprouting this so my cats can eat the grass. They love it. I rotate it around with Wheatgrass and Rye too
Summary:
 <NAME> 


Text:
 This is a very healthy dog food. Good for their digestion. Also good for small puppies. My dog eats her required amount at every feeding.
Summary:
 Healthy Dog Food 


Text:
 The Strawberry Twizzlers are my guilty pleasure - yummy. Six pounds will be around for a while with my son and I.
Summary:
 Strawberry Twizzlers - Yummy<jupyter_text>Clean and prepare the data<jupyter_code># the function gives us the option to keep_most of the characters inisde the texts and summaries, meaning
# punctuation, question marks, slashes...
# or we can set it to False, meaning we only want to keep letters and numbers like here.
processed_texts, processed_summaries, words_counted = summarizer_data_utils.preprocess_texts_and_summaries(
    raw_texts,
    raw_summaries,
    keep_most=False
)
for t,s in zip(processed_texts[:5], processed_summaries[:5]):
    print('Text\n:', t, '\n')
    print('Summary:\n', s, '\n\n\n')<jupyter_output>Text
: ['great', 'taffy', 'at', 'a', 'great', 'price', 'there', 'was', 'a', 'wide', 'assortment', 'of', 'yummy', 'taffy', 'delivery', 'was', 'very', 'quick', 'if', 'your', 'a', 'taffy', 'lover', 'this', 'is', 'a', 'deal'] 

Summary:
 ['great', 'taffy'] 



Text
: ['this', 'taffy', 'is', 'so', 'good', 'it', 'is', 'very', 'soft', 'and', 'chewy', 'the', 'flavors', 'are', 'amazing', 'i', 'would', 'definitely', 'recommend', 'you', 'buying', 'it', 'very', 'satisfying'] 

Summary:
 ['wonderful', 'tasty', 'taffy'] 



Text
: ['right', 'now', 'i', 'm', 'mostly', 'just', 'sprouting', 'this', 'so', 'my', 'cats', 'can', 'eat', 'the', 'grass', 'they', 'love', 'it', 'i', 'rotate', 'it', 'around', 'with', 'wheatgrass', 'and', 'rye', 'too'] 

Summary:
 ['yay', 'barley'] 



Text
: ['this', 'is', 'a', 'very', 'healthy', 'dog', 'food', 'good', 'for', 'their', 'digestion', 'also', 'good', 'for', 'small', 'puppies', 'my', 'dog', 'eats', 'her', 'required', 'amount', 'at', 'every', 'feeding'] 

Summary:
 ['[...]<jupyter_text>Create lookup dictsWe cannot feed our network actual words, but numbers. So we first have to create our lookup dicts, where each words gets and int value (high or low, depending on its frequency in our corpus). Those help us to later convert the texts into numbers.We also add special tokens. EndOfSentence and StartOfSentence are crucial for the Seq2Seq model we later use.Pad token, because all summaries and texts in a batch need to have the same length, pad token helps us do that.So we need 2 lookup dicts: - From word to index  - from index to word.<jupyter_code>specials = ["<EOS>", "<SOS>","<PAD>","<UNK>"]
word2ind, ind2word,  missing_words = summarizer_data_utils.create_word_inds_dicts(words_counted,
                                                                       specials = specials)
print(len(word2ind), len(ind2word), len(missing_words))<jupyter_output>25067 25067 0<jupyter_text>Pretrained embeddingsOptionally we can use pretrained word embeddings. Those have proved to increase training speed and accuracy.Here I used two different options. Either we use glove embeddings or embeddings from tf_hub.The ones from tf_hub worked better, so we use those.<jupyter_code># glove_embeddings_path = './glove.6B.300d.txt'
# embedding_matrix_save_path = './embeddings/my_embedding_github.npy'
# emb = summarizer_data_utils.create_and_save_embedding_matrix(word2ind,
#                                                              glove_embeddings_path,
#                                                              embedding_matrix_save_path)
# the embeddings from tf_hub. 
# embed = hub.Module("https://tfhub.dev/google/nnlm-en-dim128/1")
embed = hub.Module("https://tfhub.dev/google/Wiki-words-250/1")
emb = embed([key for key in word2ind.keys()])

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    sess.run(tf.tables_initializer())
    embedding = sess.run(emb)
embedding.shape
np.save('./tf_hub_embedding.npy', embedding)<jupyter_output><empty_output><jupyter_text>Convert text and summariesAs I said before we cannot feed the words directly to our network, we have to convert them to numbers first of all. This is what we do here. And we also append the SOS and EOS tokens.<jupyter_code># converts words in texts and summaries to indices
# it looks like we have to set eos here to False
converted_texts, unknown_words_in_texts = summarizer_data_utils.convert_to_inds(processed_texts,
                                                                                word2ind,
                                                                                eos = False)
converted_summaries, unknown_words_in_summaries = summarizer_data_utils.convert_to_inds(processed_summaries,
                                                                                        word2ind,
                                                                                        eos = True,
                                                                                        sos = True)
converted_texts[0]
# seems to have worked well. 
print( summarizer_data_utils.convert_inds_to_text(converted_texts[0], ind2word),
       summarizer_data_utils.convert_inds_to_text(converted_summaries[0], ind2word))<jupyter_output>['great', 'taffy', 'at', 'a', 'great', 'price', 'there', 'was', 'a', 'wide', 'assortment', 'of', 'yummy', 'taffy', 'delivery', 'was', 'very', 'quick', 'if', 'your', 'a', 'taffy', 'lover', 'this', 'is', 'a', 'deal'] ['<SOS>', 'great', 'taffy', '<EOS>']<jupyter_text>The modelNow we can build and train our model. First we define the hyperparameters we want to use. Then we create our Summarizer and call the function .build_graph(), which as the name suggests, builds the computation graph. Then we can train the model using .train()After training we can try our model using .infer()  TrainingWe can optionally use a cyclic learning rate, which we do here. I trained the model for 20 epochs and the loss was low then, but we could train it longer and would probably get better results.Unfortunately I do not have the resources to find the perfect (or right) hyperparameters, but these do pretty well.<jupyter_code># model hyperparametes
num_layers_encoder = 4
num_layers_decoder = 4
rnn_size_encoder = 512
rnn_size_decoder = 512

batch_size = 256
epochs = 200
clip = 5
keep_probability = 0.5
learning_rate = 0.0005
max_lr=0.005
learning_rate_decay_steps = 700
learning_rate_decay = 0.90


pretrained_embeddings_path = './tf_hub_embedding.npy'
summary_dir = os.path.join('./tensorboard', str('Nn_' + str(rnn_size_encoder) + '_Lr_' + str(learning_rate)))


use_cyclic_lr = True
inference_targets=True

len(converted_summaries)
round(78862*0.9)
# build graph and train the model 
summarizer_model_utils.reset_graph()
summarizer = Summarizer.Summarizer(word2ind,
                                   ind2word,
                                   save_path='./models/amazon/my_model',
                                   mode='TRAIN',
                                   num_layers_encoder = num_layers_encoder,
                                   num_layers_decoder = num_layers_decoder,
                                   rnn_size_encoder = rnn_size_encoder,
                                   rnn_size_decoder = rnn_size_decoder,
                                   batch_size = batch_size,
                                   clip = clip,
                                   keep_probability = keep_probability,
                                   learning_rate = learning_rate,
                                   max_lr=max_lr,
                                   learning_rate_decay_steps = learning_rate_decay_steps,
                                   learning_rate_decay = learning_rate_decay,
                                   epochs = epochs,
                                   pretrained_embeddings_path = pretrained_embeddings_path,
                                   use_cyclic_lr = use_cyclic_lr,
                                   summary_dir = summary_dir)           

summarizer.build_graph()
summarizer.train(converted_texts[:70976], 
                 converted_summaries[:70976],
                 validation_inputs=converted_texts[70976:],
                 validation_targets=converted_summaries[70976:])


# hidden training output.
# both train and validation loss decrease nicely.<jupyter_output><empty_output><jupyter_text>InferenceNow we can use our trained model to create summaries.<jupyter_code>summarizer_model_utils.reset_graph()
summarizer = Summarizer.Summarizer(word2ind,
                                   ind2word,
                                   './models/amazon/my_model',
                                   'INFER',
                                   num_layers_encoder = num_layers_encoder,
                                   num_layers_decoder = num_layers_decoder,
                                   batch_size = len(converted_texts[:50]),
                                   clip = clip,
                                   keep_probability = 1.0,
                                   learning_rate = 0.0,
                                   beam_width = 5,
                                   rnn_size_encoder = rnn_size_encoder,
                                   rnn_size_decoder = rnn_size_decoder,
                                   inference_targets = True,
                                   pretrained_embeddings_path = pretrained_embeddings_path)

summarizer.build_graph()
preds = summarizer.infer(converted_texts[:50],
                         restore_path =  './models/amazon/my_model',
                         targets = converted_summaries[:50])

# show results
summarizer_model_utils.sample_results(preds,
                                      ind2word,
                                      word2ind,
                                      converted_summaries[:50],
                                      converted_texts[:50])<jupyter_output>----------------------------------------------------------------------------------------------------
Actual Text:
great taffy at a great price there was a wide assortment of yummy taffy delivery was very quick if your a taffy lover this is a deal

Actual Summary:
great taffy

Created Summary:
taffy taffy taffy





 ----------------------------------------------------------------------------------------------------
Actual Text:
this taffy is so good it is very soft and chewy the flavors are amazing i would definitely recommend you buying it very satisfying

Actual Summary:
wonderful tasty taffy

Created Summary:
taffy taffy





 ----------------------------------------------------------------------------------------------------
Actual Text:
right now i m mostly just sprouting this so my cats can eat the grass they love it i rotate it around with wheatgrass and rye too

Actual Summary:
yay barley

Created Summary:
yay barley





 ---------------------------------------------------[...]
{'fraction_non_alphanumeric': 0.11746987951807229, 'fraction_numerical': 0.01069277108433735, 'mean_word_length': 2.291449814126394, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 35, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 61, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '243769', 'license': 'MIT', 'n_tokens_mistral': 3903, 'n_tokens_neox': 3671, 'n_words': 1215, 'path': 'summarizer_amazon_reviews.ipynb', 'repo_name': 'zzzZLP/TextSummarization', 'size': '13284'}