<jupyter_start><jupyter_text>In this notebook, we will build an abstractive based text summarizer using deep learning from the scratch in python using kerasI recommend you to go through the article over [here](https://www.analyticsvidhya.com/blog/2019/06/comprehensive-guide-text-summarization-using-deep-learning-python/) to cover all the concepts which is required to build our own summarizer Understanding the Problem StatementCustomer reviews can often be long and descriptive. Analyzing these reviews manually, as you can imagine, is really time-consuming. This is where the brilliance of Natural Language Processing can be applied to generate a summary for long reviews.We will be working on a really cool dataset. Our objective here is to generate a summary for the Amazon Fine Food reviews using the abstraction-based approach we learned about above. You can download the dataset from[ here ](https://www.kaggle.com/snap/amazon-fine-food-reviews)It’s time to fire up our Jupyter notebooks! Let’s dive into the implementation details right away.Custom Attention LayerKeras does not officially support attention layer. So, we can either implement our own attention layer or use a third-party implementation. We will go with the latter option for this article. You can download the attention layer from [here](https://github.com/thushv89/attention_keras/blob/master/layers/attention.py) and copy it in a different file called attention.py.Let’s import it into our environment:<jupyter_code>import tensorflow as tf
import os
from tensorflow.python.keras.layers import Layer
from tensorflow.python.keras import backend as K
PRETRAINED_WEIGHT_NAME=""
on_colab=True
base_dir="gdrive/My Drive/News Summary"
glove_dir="glove.6B.300d.txt"
data_dir=r"wikihowSep.csv"
VERBOSE=1
EPOCH=20
BATCHSIZE=32
try:
    from google.colab import drive
    drive.mount("/content/gdrive")
except ImportError:
    print("Running on Local")
    on_colab=False
    base_dir=os.getcwd()
    glove_dir=r"C:\Users\fflea\PycharmProjects\CSC790_Deep_Learning\Image-Captioning\glove.6B.200d.txt"
    data_dir=r"E:\Wikihow\wikihowSep.csv"
    VERBOSE=2
if on_colab:    
    !python -m nltk.downloader stopwords
    if not os.path.exists(base_dir):
        os.mkdir(base_dir)
class AttentionLayer(Layer): 
  def init(self, kwargs): super(AttentionLayer, self).init(kwargs)

  def build(self, input_shape):
      assert isinstance(input_shape, list)
      # Create a trainable weight variable for this layer.
  
      self.W_a = self.add_weight(name='W_a',
                                shape=tf.TensorShape((input_shape[0][2], input_shape[0][2])),
                                initializer='uniform',
                                trainable=True)
      self.U_a = self.add_weight(name='U_a',
                                shape=tf.TensorShape((input_shape[1][2], input_shape[0][2])),
                                initializer='uniform',
                                trainable=True)
      self.V_a = self.add_weight(name='V_a',
                                shape=tf.TensorShape((input_shape[0][2], 1)),
                                initializer='uniform',
                                trainable=True)
  
      super(AttentionLayer, self).build(input_shape)  # Be sure to call this at the end
  
  def call(self, inputs, verbose=False):
      """
      inputs: [encoder_output_sequence, decoder_output_sequence]
      """
      assert type(inputs) == list
      encoder_out_seq, decoder_out_seq = inputs
      if verbose:
          print('encoder_out_seq>', encoder_out_seq.shape)
          print('decoder_out_seq>', decoder_out_seq.shape)
  
      def energy_step(inputs, states):
          """ Step function for computing energy for a single decoder state
          inputs: (batchsize * 1 * de_in_dim)
          states: (batchsize * 1 * de_latent_dim)
          """
  
          assert_msg = "States must be an iterable. Got {} of type {}".format(states, type(states))
          assert isinstance(states, list) or isinstance(states, tuple), assert_msg
  
          """ Some parameters required for shaping tensors"""
          en_seq_len, en_hidden = encoder_out_seq.shape[1], encoder_out_seq.shape[2]
          de_hidden = inputs.shape[-1]
  
          """ Computing S.Wa where S=[s0, s1, ..., si]"""
          # <= batch size * en_seq_len * latent_dim
          W_a_dot_s = K.dot(encoder_out_seq, self.W_a)
  
          """ Computing hj.Ua """
          U_a_dot_h = K.expand_dims(K.dot(inputs, self.U_a), 1)  # <= batch_size, 1, latent_dim
          if verbose:
              print('Ua.h>', U_a_dot_h.shape)
  
          """ tanh(S.Wa + hj.Ua) """
          # <= batch_size*en_seq_len, latent_dim
          Ws_plus_Uh = K.tanh(W_a_dot_s + U_a_dot_h)
          if verbose:
              print('Ws+Uh>', Ws_plus_Uh.shape)
  
          """ softmax(va.tanh(S.Wa + hj.Ua)) """
          # <= batch_size, en_seq_len
          e_i = K.squeeze(K.dot(Ws_plus_Uh, self.V_a), axis=-1)
          # <= batch_size, en_seq_len
          e_i = K.softmax(e_i)
  
          if verbose:
              print('ei>', e_i.shape)
  
          return e_i, [e_i]
  
      def context_step(inputs, states):
          """ Step function for computing ci using ei """
  
          assert_msg = "States must be an iterable. Got {} of type {}".format(states, type(states))
          assert isinstance(states, list) or isinstance(states, tuple), assert_msg
  
          # <= batch_size, hidden_size
          c_i = K.sum(encoder_out_seq * K.expand_dims(inputs, -1), axis=1)
          if verbose:
              print('ci>', c_i.shape)
          return c_i, [c_i]
  
      fake_state_c = K.sum(encoder_out_seq, axis=1)
      fake_state_e = K.sum(encoder_out_seq, axis=2)  # <= (batch_size, enc_seq_len, latent_dim
  
      """ Computing energy outputs """
      # e_outputs => (batch_size, de_seq_len, en_seq_len)
      last_out, e_outputs, _ = K.rnn(
          energy_step, decoder_out_seq, [fake_state_e],
      )
  
      """ Computing context vectors """
      last_out, c_outputs, _ = K.rnn(
          context_step, e_outputs, [fake_state_c],
      )
  
      return c_outputs, e_outputs
  
  def compute_output_shape(self, input_shape):
      """ Outputs produced by the layer """
      return [
          tf.TensorShape((input_shape[1][0], input_shape[1][1], input_shape[1][2])),
          tf.TensorShape((input_shape[1][0], input_shape[1][1], input_shape[0][1]))
      ]<jupyter_output><empty_output><jupyter_text>Import the Libraries<jupyter_code>import numpy as np
import pandas as pd 
import re
from bs4 import BeautifulSoup
import io
import json
from keras.preprocessing.text import Tokenizer, tokenizer_from_json
from keras.preprocessing.sequence import pad_sequences
from nltk.corpus import stopwords
from tensorflow.keras.layers import Input, LSTM, Embedding, Dense, Concatenate, TimeDistributed
from tensorflow.keras.models import Model,save_model, load_model
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
from tensorflow.keras.utils import plot_model
import warnings
from matplotlib import pyplot
pd.set_option("display.max_colwidth", 200)
warnings.filterwarnings("ignore")<jupyter_output>Using TensorFlow backend.<jupyter_text>Read the datasetThis dataset consists of reviews of fine foods from Amazon. The data spans a period of more than 10 years, including all ~500,000 reviews up to October 2012. These reviews include product and user information, ratings, plain text review, and summary. It also includes reviews from all other Amazon categories.We’ll take a sample of 100,000 reviews to reduce the training time of our model. Feel free to use the entire dataset for training your model if your machine has that kind of computational power.<jupyter_code>nrows=100000
try:
  data=pd.read_csv(os.path.join(base_dir,data_dir),nrows=nrows)
except FileNotFoundError:
  data=pd.read_csv(data_dir,nrows=nrows)
if "content" in data.columns:
    data.rename(columns = {"content":"article"}, inplace = True)
if "Summary" in data.columns:
    data.rename(columns = {"Summary":"summary"}, inplace = True)
if "headline" in data.columns:
    data.rename(columns = {"headline":"summary"}, inplace = True)
if "Text" in data.columns:
    data.rename(columns = {"Text":"article"}, inplace = True)
if "text" in data.columns:
    data.rename(columns = {"text":"article"}, inplace = True)
print(data.head(5))<jupyter_output>overview  ...                        title
0   So you're a new or aspiring artist and your creativity has spawned something  unique  and interesting. Now you're ready to display and sell this creation and perhaps make some money, but you're v...  ...  How to Sell Fine Art Online
1   If you want to be well-read, then, in the words of <NAME>, you'll have to "Read, read, read. Read everything..." You can start at the very beginning, or just make your way down an eclec...  ...          How to Be Well Read
2   So you're a new or aspiring artist and your creativity has spawned something  unique  and interesting. Now you're ready to display and sell this creation and perhaps make some money, but you're v...  ...  How to Sell Fine Art Online
3   So you're a new or aspiring artist and your crea[...]<jupyter_text>Drop Duplicates and NA values<jupyter_code>data.drop_duplicates(subset=["article"],inplace=True)#dropping duplicates
data = data[pd.notnull(data["article"])]
data = data[pd.notnull(data["summary"])]
#data.dropna(axis=0,inplace=True)#dropping na<jupyter_output><empty_output><jupyter_text>Information about datasetLet us look at datatypes and shape of the dataset<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 95979 entries, 0 to 99999
Data columns (total 5 columns):
 #   Column        Non-Null Count  Dtype 
---  ------        --------------  ----- 
 0   overview      95672 non-null  object
 1   summary       95979 non-null  object
 2   article       95979 non-null  object
 3   sectionLabel  95832 non-null  object
 4   title         95979 non-null  object
dtypes: object(5)
memory usage: 4.4+ MB<jupyter_text>PreprocessingPerforming basic preprocessing steps is very important before we get to the model building part. Using messy and uncleaned text data is a potentially disastrous move. So in this step, we will drop all the unwanted symbols, characters, etc. from the text that do not affect the objective of our problem.Here is the dictionary that we will use for expanding the contractions:<jupyter_code>contraction_mapping = {"ain't": "is not", "aren't": "are not","can't": "cannot", "'cause": "because", "could've": "could have", "couldn't": "could not",
                           "didn't": "did not",  "doesn't": "does not", "don't": "do not", "hadn't": "had not", "hasn't": "has not", "haven't": "have not",
                           "he'd": "he would","he'll": "he will", "he's": "he is", "how'd": "how did", "how'd'y": "how do you", "how'll": "how will", "how's": "how is",
                           "I'd": "I would", "I'd've": "I would have", "I'll": "I will", "I'll've": "I will have","I'm": "I am", "I've": "I have", "i'd": "i would",
                           "i'd've": "i would have", "i'll": "i will",  "i'll've": "i will have","i'm": "i am", "i've": "i have", "isn't": "is not", "it'd": "it would",
                           "it'd've": "it would have", "it'll": "it will", "it'll've": "it will have","it's": "it is", "let's": "let us", "ma'am": "madam",
                           "mayn't": "may not", "might've": "might have","mightn't": "might not","mightn't've": "might not have", "must've": "must have",
                           "mustn't": "must not", "mustn't've": "must not have", "needn't": "need not", "needn't've": "need not have","o'clock": "of the clock",
                           "oughtn't": "ought not", "oughtn't've": "ought not have", "shan't": "shall not", "sha'n't": "shall not", "shan't've": "shall not have",
                           "she'd": "she would", "she'd've": "she would have", "she'll": "she will", "she'll've": "she will have", "she's": "she is",
                           "should've": "should have", "shouldn't": "should not", "shouldn't've": "should not have", "so've": "so have","so's": "so as",
                           "this's": "this is","that'd": "that would", "that'd've": "that would have", "that's": "that is", "there'd": "there would",
                           "there'd've": "there would have", "there's": "there is", "here's": "here is","they'd": "they would", "they'd've": "they would have",
                           "they'll": "they will", "they'll've": "they will have", "they're": "they are", "they've": "they have", "to've": "to have",
                           "wasn't": "was not", "we'd": "we would", "we'd've": "we would have", "we'll": "we will", "we'll've": "we will have", "we're": "we are",
                           "we've": "we have", "weren't": "were not", "what'll": "what will", "what'll've": "what will have", "what're": "what are",
                           "what's": "what is", "what've": "what have", "when's": "when is", "when've": "when have", "where'd": "where did", "where's": "where is",
                           "where've": "where have", "who'll": "who will", "who'll've": "who will have", "who's": "who is", "who've": "who have",
                           "why's": "why is", "why've": "why have", "will've": "will have", "won't": "will not", "won't've": "will not have",
                           "would've": "would have", "wouldn't": "would not", "wouldn't've": "would not have", "y'all": "you all",
                           "y'all'd": "you all would","y'all'd've": "you all would have","y'all're": "you all are","y'all've": "you all have",
                           "you'd": "you would", "you'd've": "you would have", "you'll": "you will", "you'll've": "you will have",
                           "you're": "you are", "you've": "you have"}<jupyter_output><empty_output><jupyter_text>We will perform the below preprocessing tasks for our data:1.Convert everything to lowercase2.Remove HTML tags3.Contraction mapping4.Remove (‘s)5.Remove any text inside the parenthesis ( )6.Eliminate punctuations and special characters7.Remove stopwords8.Remove short wordsLet’s define the function:<jupyter_code>stop_words = set(stopwords.words('english')) 

def text_cleaner(text,num):
    newString = text.lower()
    newString = BeautifulSoup(newString, "html.parser").text
    newString = re.sub(r'\([^)]*\)', '', newString)
    newString = re.sub('"','', newString)
    newString = ' '.join([contraction_mapping[t] if t in contraction_mapping else t for t in newString.split(" ")])    
    newString = re.sub(r"'s\b","",newString)
    newString = re.sub("[^a-zA-Z]", " ", newString) 
    newString = re.sub('[m]{2,}', 'mm', newString)
    if(num==0):
        tokens = [w for w in newString.split() if not w in stop_words]
    else:
        tokens=newString.split()
    long_words=[]
    for i in tokens:
        if len(i)>1:                                                 #removing short word
            long_words.append(i)   
    return (" ".join(long_words)).strip()
#call the function
cleaned_text = []
for t in data["article"]:
    cleaned_text.append(text_cleaner(t,0))<jupyter_output><empty_output><jupyter_text>Let us look at the first five preprocessed reviews<jupyter_code>cleaned_text[:5]  
#call the function
cleaned_summary = []
for t in data["summary"]:
    cleaned_summary.append(text_cleaner(t,1))<jupyter_output><empty_output><jupyter_text>Let us look at the first 10 preprocessed summaries<jupyter_code>cleaned_summary[:10]
data['cleaned_text']=cleaned_text
data['cleaned_summary']=cleaned_summary<jupyter_output><empty_output><jupyter_text>Understanding the distribution of the sequencesHere, we will analyze the length of the reviews and the summary to get an overall idea about the distribution of length of the text. This will help us fix the maximum length of the sequence:<jupyter_code>import matplotlib.pyplot as plt

text_word_count = []
summary_word_count = []

# populate the lists with sentence lengths
for i in data['cleaned_text']:
      text_word_count.append(len(i.split()))

for i in data['cleaned_summary']:
      summary_word_count.append(len(i.split()))

length_df = pd.DataFrame({'text':text_word_count, 'summary':summary_word_count})

length_df.hist(bins = 30)
plt.show()<jupyter_output><empty_output><jupyter_text>Interesting. We can fix the maximum length of the summary to 8 since that seems to be the majority summary length.Let us understand the proportion of the length of summaries below 8<jupyter_code>cnt=0
for i in data['cleaned_summary']:
    if(len(i.split())<15):
        cnt=cnt+1
print(cnt/len(data['cleaned_summary']))<jupyter_output>0.939465924837725<jupyter_text>We observe that 94% of the summaries have length below 8. So, we can fix maximum length of summary to 8.Let us fix the maximum length of review to 30<jupyter_code>max_text_len=150
max_summary_len=15<jupyter_output><empty_output><jupyter_text>Let us select the reviews and summaries whose length falls below or equal to **max_text_len** and **max_summary_len**<jupyter_code>cleaned_text =np.array(data['cleaned_text'])
cleaned_summary=np.array(data['cleaned_summary'])

short_text=[]
short_summary=[]

for i in range(len(cleaned_text)):
    if(len(cleaned_summary[i].split())<=max_summary_len and len(cleaned_text[i].split())<=max_text_len):
        short_text.append(cleaned_text[i])
        short_summary.append(cleaned_summary[i])
        
df=pd.DataFrame({"article":short_text,"summary":short_summary})<jupyter_output><empty_output><jupyter_text>Remember to add the **START** and **END** special tokens at the beginning and end of the summary. Here, I have chosen **sostok** and **eostok** as START and END tokens**Note:** Be sure that the chosen special tokens never appear in the summary<jupyter_code>df["summary"] = df["summary"].apply(lambda x : 'sostok '+ x + ' eostok')<jupyter_output><empty_output><jupyter_text>We are getting closer to the model building part. Before that, we need to split our dataset into a training and validation set. We’ll use 90% of the dataset as the training data and evaluate the performance on the remaining 10% (holdout set):<jupyter_code>from sklearn.model_selection import train_test_split
x_tr,x_val,y_tr,y_val=train_test_split(np.array(df["article"]),np.array(df["summary"]),test_size=0.1,random_state=0,shuffle=True)<jupyter_output><empty_output><jupyter_text>Preparing the TokenizerA tokenizer builds the vocabulary and converts a word sequence to an integer sequence. Go ahead and build tokenizers for text and summary:Text Tokenizer<jupyter_code>#prepare a tokenizer for reviews on training data
x_tokenizer = Tokenizer() 
x_tokenizer.fit_on_texts(list(x_tr))<jupyter_output><empty_output><jupyter_text>Rarewords and its CoverageLet us look at the proportion rare words and its total coverage in the entire textHere, I am defining the threshold to be 4 which means word whose count is below 4 is considered as a rare word<jupyter_code>thresh=4

cnt=0
tot_cnt=0
freq=0
tot_freq=0

for key,value in x_tokenizer.word_counts.items():
    tot_cnt=tot_cnt+1
    tot_freq=tot_freq+value
    if(value<thresh):
        cnt=cnt+1
        freq=freq+value
    
print("% of rare words in vocabulary:",(cnt/tot_cnt)*100)
print("Total Coverage of rare words:",(freq/tot_freq)*100)<jupyter_output>% of rare words in vocabulary: 48.19134078212291
Total Coverage of rare words: 0.8621581033506487<jupyter_text>**Remember**:* **tot_cnt** gives the size of vocabulary (which means every unique words in the text) *   **cnt** gives me the no. of rare words whose count falls below threshold*  **tot_cnt - cnt** gives me the top most common words Let us define the tokenizer with top most common words for reviews.<jupyter_code>#prepare a tokenizer for reviews on training data
x_tokenizer = Tokenizer(num_words=tot_cnt-cnt) 
x_tokenizer.fit_on_texts(list(x_tr))

#convert text sequences into integer sequences
x_tr_seq    =   x_tokenizer.texts_to_sequences(x_tr) 
x_val_seq   =   x_tokenizer.texts_to_sequences(x_val)

#padding zero upto maximum length
x_tr    =   pad_sequences(x_tr_seq,  maxlen=max_text_len, padding='post')
x_val   =   pad_sequences(x_val_seq, maxlen=max_text_len, padding='post')

#size of vocabulary ( +1 for padding token)
x_voc   =  len(x_tokenizer.word_index) + 1
x_voc<jupyter_output><empty_output><jupyter_text>Summary Tokenizer<jupyter_code>#prepare a tokenizer for reviews on training data
y_tokenizer = Tokenizer()   
y_tokenizer.fit_on_texts(list(y_tr))<jupyter_output><empty_output><jupyter_text>Rarewords and its CoverageLet us look at the proportion rare words and its total coverage in the entire summaryHere, I am defining the threshold to be 6 which means word whose count is below 6 is considered as a rare word<jupyter_code>thresh=6

cnt=0
tot_cnt=0
freq=0
tot_freq=0

for key,value in y_tokenizer.word_counts.items():
    tot_cnt=tot_cnt+1
    tot_freq=tot_freq+value
    if(value<thresh):
        cnt=cnt+1
        freq=freq+value
    
print("% of rare words in vocabulary:",(cnt/tot_cnt)*100)
print("Total Coverage of rare words:",(freq/tot_freq)*100)<jupyter_output>% of rare words in vocabulary: 66.45307199396909
Total Coverage of rare words: 3.3309283114314208<jupyter_text>Let us define the tokenizer with top most common words for summary.<jupyter_code>#prepare a tokenizer for reviews on training data
y_tokenizer = Tokenizer(num_words=tot_cnt-cnt) 
y_tokenizer.fit_on_texts(list(y_tr))

#convert text sequences into integer sequences
y_tr_seq    =   y_tokenizer.texts_to_sequences(y_tr) 
y_val_seq   =   y_tokenizer.texts_to_sequences(y_val) 

#padding zero upto maximum length
y_tr    =   pad_sequences(y_tr_seq, maxlen=max_summary_len, padding='post')
y_val   =   pad_sequences(y_val_seq, maxlen=max_summary_len, padding='post')

#size of vocabulary
y_voc  =   len(y_tokenizer.word_index) +1<jupyter_output><empty_output><jupyter_text>Let us check whether word count of start token is equal to length of the training data<jupyter_code>y_tokenizer.word_counts['sostok'],len(y_tr)<jupyter_output><empty_output><jupyter_text>Here, I am deleting the rows that contain only **START** and **END** tokens<jupyter_code>ind=[]
for i in range(len(y_tr)):
    cnt=0
    for j in y_tr[i]:
        if j!=0:
            cnt=cnt+1
    if(cnt==2):
        ind.append(i)

y_tr=np.delete(y_tr,ind, axis=0)
x_tr=np.delete(x_tr,ind, axis=0)
ind=[]
for i in range(len(y_val)):
    cnt=0
    for j in y_val[i]:
        if j!=0:
            cnt=cnt+1
    if(cnt==2):
        ind.append(i)

y_val=np.delete(y_val,ind, axis=0)
x_val=np.delete(x_val,ind, axis=0)<jupyter_output><empty_output><jupyter_text>Model buildingWe are finally at the model building part. But before we do that, we need to familiarize ourselves with a few terms which are required prior to building the model.**Return Sequences = True**: When the return sequences parameter is set to True, LSTM produces the hidden state and cell state for every timestep**Return State = True**: When return state = True, LSTM produces the hidden state and cell state of the last timestep only**Initial State**: This is used to initialize the internal states of the LSTM for the first timestep**Stacked LSTM**: Stacked LSTM has multiple layers of LSTM stacked on top of each other. This leads to a better representation of the sequence. I encourage you to experiment with the multiple layers of the LSTM stacked on top of each other (it’s a great way to learn this)Here, we are building a 3 stacked LSTM for the encoder:<jupyter_code>latent_dim = 500
embedding_dim=300
# Preparing GloVe
embeddings_index = {}
try:
  f = open(glove_dir,encoding="utf8")
except FileNotFoundError:
  f = open(os.path.join(base_dir,glove_dir),encoding="utf8")
for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype='float32')
    embeddings_index[word] = coefs
f.close()
embedding_matrix = np.zeros((x_voc, embedding_dim),dtype='float32')
for word,i in x_tokenizer.word_index.items():
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
    # Words not found in glove will be zeros
        embedding_matrix[i] = embedding_vector
from keras import backend as K 
K.clear_session()


# Encoder
encoder_inputs = Input(shape=(max_text_len,))

#embedding layer
enc_emb =  Embedding(x_voc, embedding_dim,weights=[embedding_matrix],
input_length=max_text_len, trainable=False,  name='embedding_layer')(encoder_inputs)

#encoder lstm 1
encoder_lstm1 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)
encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)

#encoder lstm 2
encoder_lstm2 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)
encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)

#encoder lstm 3
encoder_lstm3=LSTM(latent_dim, return_state=True, return_sequences=True,dropout=0.4,recurrent_dropout=0.4)
encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)

# Set up the decoder, using `encoder_states` as initial state.
decoder_inputs = Input(shape=(None,))

#embedding layer
dec_emb_layer = Embedding(y_voc, embedding_dim,trainable=True)
dec_emb = dec_emb_layer(decoder_inputs)

decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True,dropout=0.4,recurrent_dropout=0.2)
decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(dec_emb,initial_state=[state_h, state_c])

# Attention layer
attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])

# Concat attention input and decoder LSTM output
decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])

#dense layer
decoder_dense =  TimeDistributed(Dense(y_voc, activation='softmax'))
decoder_outputs = decoder_dense(decoder_concat_input)

# Define the model 
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

model.summary()<jupyter_output>WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:Entity <bound method AttentionLayer.call of <__main__.AttentionLayer object at 0x7f45a8560160>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSIT[...]<jupyter_text>I am using sparse categorical cross-entropy as the loss function since it converts the integer sequence to a one-hot vector on the fly. This overcomes any memory issues.<jupyter_code>plot_model(model,os.path.join(base_dir,"plot_model.png"))
model.compile(optimizer='RMSProp', loss='sparse_categorical_crossentropy')<jupyter_output><empty_output><jupyter_text>Remember the concept of early stopping? It is used to stop training the neural network at the right time by monitoring a user-specified metric. Here, I am monitoring the validation loss (val_loss). Our model will stop training once the validation loss increases:<jupyter_code>#if os.path.exists(os.path.join(base_dir,"x_tokenizer.json")):
#  with open(os.path.join(base_dir,"x_tokenizer.json")) as fx:
#      data = json.load(fx)
#      x_tokenizer_json = tokenizer_from_json(data)
#  print("Loaded Tokenizer X")
#else:
x_tokenizer_json = x_tokenizer.to_json()
with io.open(os.path.join(base_dir,"x_tokenizer.json"), 'w', encoding='utf-8') as fx:
    fx.write(json.dumps(x_tokenizer_json, ensure_ascii=False))

#if os.path.exists(os.path.join(base_dir,"y_tokenizer.json")):
#  with open(os.path.join(base_dir,"y_tokenizer.json")) as fy:
#      data = json.load(fy)
#      y_tokenizer_json = tokenizer_from_json(data)
#  print("Loaded Tokenizer Y")
#else:
y_tokenizer_json = y_tokenizer.to_json()
with io.open(os.path.join(base_dir,"y_tokenizer.json"), 'w', encoding='utf-8') as fy:
    fy.write(json.dumps(y_tokenizer_json, ensure_ascii=False))
es = EarlyStopping(monitor='val_loss', mode='min', verbose=VERBOSE,patience=3)

ckp_path=os.path.join(base_dir,"Checkpoints")
if not os.path.exists(ckp_path):
  os.mkdir(ckp_path)
ckp = ModelCheckpoint((os.path.join(ckp_path, "weight.{epoch:02d}-{loss:.3f}-{val_loss:.3f}.h5")))<jupyter_output><empty_output><jupyter_text>We’ll train the model on a batch size of 128 and validate it on the holdout set (which is 10% of our dataset):<jupyter_code>if PRETRAINED_WEIGHT_NAME!="" and os.path.exists(os.path.join(base_dir,PRETRAINED_WEIGHT_NAME)):
    model=load_model(os.path.join(base_dir,PRETRAINED_WEIGHT_NAME))
    print("Load pretrained model")
history=model.fit([x_tr,y_tr[:,:-1]], y_tr.reshape(y_tr.shape[0],y_tr.shape[1], 1)[:,1:] ,
                  verbose=VERBOSE,epochs=EPOCH,
                  callbacks=[es,ckp],
                  batch_size=BATCHSIZE,
                  validation_data=([x_val,y_val[:,:-1]], y_val.reshape(y_val.shape[0],y_val.shape[1], 1)[:,1:]))<jupyter_output>WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Train on 81403 samples, validate on 9039 samples
Epoch 1/20
81403/81403 [==============================] - 2196s 27ms/sample - loss: 2.3044 - val_loss: 2.0829
Epoch 2/20
81403/81403 [==============================] - 2193s 27ms/sample - loss: 2.0535 - val_loss: 1.9664
Epoch 3/20
81403/81403 [==============================] - 2159s 27ms/sample - loss: 1.9575 - val_loss: 1.9094
Epoch 4/20
81403/81403 [==============================] - 2176s 27ms/sample - loss: 1.9049 - val_loss: 1.8754
Epoch 5/20
81403/81403 [==============================] - 2165s 27ms/sample - loss: 1.8716 - val_loss: 1.8553
Epoch 6/20
81403/81403 [==============================] - 2135s 26ms/sample - loss: 1.8501 - val_loss: 1.841[...]<jupyter_text>Understanding the Diagnostic plotNow, we will plot a few diagnostic plots to understand the behavior of the model over time:<jupyter_code>try:
    pyplot.plot(history.history['loss'], label='train')
    pyplot.plot(history.history['val_loss'], label='test')
    pyplot.legend()
    pyplot.show()
except NameError:
    print("Did not finish training")
save_model(model,os.path.join(base_dir,"trained.h5"))
reverse_target_word_index=y_tokenizer.index_word
reverse_source_word_index=x_tokenizer.index_word
target_word_index=y_tokenizer.word_index<jupyter_output><empty_output><jupyter_text>InferenceSet up the inference for the encoder and decoder:<jupyter_code># Encode the input sequence to get the feature vector
encoder_model = Model(inputs=encoder_inputs,outputs=[encoder_outputs, state_h, state_c])

# Decoder setup
# Below tensors will hold the states of the previous time step
decoder_state_input_h = Input(shape=(latent_dim,))
decoder_state_input_c = Input(shape=(latent_dim,))
decoder_hidden_state_input = Input(shape=(max_text_len,latent_dim))

# Get the embeddings of the decoder sequence
dec_emb2= dec_emb_layer(decoder_inputs) 
# To predict the next word in the sequence, set the initial states to the states from the previous time step
decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])

#attention inference
attn_out_inf, attn_states_inf = attn_layer([decoder_hidden_state_input, decoder_outputs2])
decoder_inf_concat = Concatenate(axis=-1, name='concat')([decoder_outputs2, attn_out_inf])

# A dense softmax layer to generate prob dist. over the target vocabulary
decoder_outputs2 = decoder_dense(decoder_inf_concat) 

# Final decoder model
decoder_model = Model(
    [decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],
    [decoder_outputs2] + [state_h2, state_c2])<jupyter_output><empty_output><jupyter_text>We are defining a function below which is the implementation of the inference process (which we covered [here](https://www.analyticsvidhya.com/blog/2019/06/comprehensive-guide-text-summarization-using-deep-learning-python/)):<jupyter_code>def decode_sequence(input_seq):
    # Encode the input as state vectors.
    e_out, e_h, e_c = encoder_model.predict(input_seq)
    
    # Generate empty target sequence of length 1.
    target_seq = np.zeros((1,1))
    
    # Populate the first word of target sequence with the start word.
    target_seq[0, 0] = target_word_index['sostok']

    stop_condition = False
    decoded_sentence = ''
    while not stop_condition:
      
        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])

        # Sample a token
        sampled_token_index = np.argmax(output_tokens[0, -1, :])
        sampled_token = reverse_target_word_index[sampled_token_index]
        
        if(sampled_token!='eostok'):
            decoded_sentence += ' '+sampled_token

        # Exit condition: either hit max length or find stop word.
        if (sampled_token == 'eostok'  or len(decoded_sentence.split()) >= (max_summary_len-1)):
            stop_condition = True

        # Update the target sequence (of length 1).
        target_seq = np.zeros((1,1))
        target_seq[0, 0] = sampled_token_index

        # Update internal states
        e_h, e_c = h, c

    return decoded_sentence
save_model(encoder_model,os.path.join(base_dir,"encoder_model.h5")) 
save_model(decoder_model,os.path.join(base_dir,"decoder_model.h5"))<jupyter_output><empty_output><jupyter_text>Let us define the functions to convert an integer sequence to a word sequence for summary as well as the reviews:<jupyter_code>def seq2summary(input_seq):
    newString=''
    for i in input_seq:
        if((i!=0 and i!=target_word_index['sostok']) and i!=target_word_index['eostok']):
            newString=newString+reverse_target_word_index[i]+' '
    return newString

def seq2text(input_seq):
    newString=''
    for i in input_seq:
        if(i!=0):
            newString=newString+reverse_source_word_index[i]+' '
    return newString<jupyter_output><empty_output><jupyter_text>Here are a few summaries generated by the model:<jupyter_code>for i in range(0,100):
    print("Review:",seq2text(x_tr[i]))
    print("Original summary:",seq2summary(y_tr[i]))
    print("Predicted summary:",decode_sequence(x_tr[i].reshape(1,max_text_len)))
    print("\n")
if on_colab:
  while True:pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12235026319533361, 'fraction_numerical': 0.018693982074263763, 'mean_word_length': 4.257442034405385, 'pattern_counts': {'":': 127, '<': 137, '<?xml version=': 0, '>': 134, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '295427', 'license': 'MIT', 'n_tokens_mistral': 12028, 'n_tokens_neox': 11302, 'n_words': 3124, 'path': 'NLP/How_to_build_own_text_summarizer_using_deep_learning_1.ipynb', 'repo_name': 'Hieu-Minh-Phung/Data-Science-Projects', 'size': '35155'}