<jupyter_start><jupyter_text>RECURRENT NETWORKS and LSTM IN DEEP LEARNING Applying Recurrent Neural Networks/LSTM for Language ModellingHello and welcome to this part. In this notebook, we will go over the topic of what Language Modelling is and create a Recurrent Neural Network model based on the Long Short-Term Memory unit to train and be benchmarked by the Penn Treebank. By the end of this notebook, you should be able to understand how TensorFlow builds and executes a RNN model for Language Modelling. --------  The ObjectiveBy now, you should have an understanding of how Recurrent Networks work -- a specialized model to process sequential data by keeping track of the "state" or context. In this notebook, we go over a TensorFlow code snippet for creating a model focused on **Language Modelling** -- a very relevant task that is the cornerstone of many different linguistic problems such as **Speech Recognition, Machine Translation and Image Captioning**. For this, we will be using the Penn Treebank, which is an often-used dataset for benchmarking Language Modelling models. What exactly is Language Modelling?Language Modelling, to put it simply, **is the task of assigning probabilities to sequences of words**. This means that, given a context of one or a few words in the language the model was trained on, the model should have a knowledge of what are the most probable words or sequence of words for the sentence. Language Modelling is one of the tasks under Natural Language Processing, and one of the most important.*Example of a sentence being predicted*In this example, one can see the predictions for the next word of a sentence, given the context "This is an". As you can see, this boils down to a sequential data analysis task -- you are given a word or a sequence of words (the input data), and, given the context (the state), you need to find out what is the next word (the prediction). This kind of analysis is very important for language-related tasks such as **Speech Recognition, Machine Translation, Image Captioning, Text Correction** and many other very relevant problems. *The above example schematized as an RNN in execution*As the above image shows, Recurrent Network models fit this problem like a glove. Alongside LSTM and its capacity to maintain the model's state for over one thousand time steps, we have all the tools we need to undertake this problem. The goal for this notebook is to create a model that can reach **low levels of perplexity** on our desired dataset.For Language Modelling problems, **perplexity** is the way to gauge efficiency. Perplexity is simply a measure of how well a probabilistic model is able to predict its sample. A higher-level way to explain this would be saying that **low perplexity means a higher degree of trust in the predictions the model makes**. Therefore, the lower perplexity is, the better. The Penn Treebank datasetHistorically, datasets big enough for Natural Language Processing are hard to come by. This is in part due to the necessity of the sentences to be broken down and tagged with a certain degree of correctness -- or else the models trained on it won't be able to be correct at all. This means that we need a **large amount of data, annotated by or at least corrected by humans**. This is, of course, not an easy task at all.The Penn Treebank, or PTB for short, is a dataset maintained by the University of Pennsylvania. It is *huge* -- there are over **four million and eight hundred thousand** annotated words in it, all corrected by humans. It is composed of many different sources, from abstracts of Department of Energy papers to texts from the Library of America. Since it is verifiably correct and of such a huge size, the Penn Treebank has been used time and time again as a benchmark dataset for Language Modelling.The dataset is divided in different kinds of annotations, such as Piece-of-Speech, Syntactic and Semantic skeletons. For this example, we will simply use a sample of clean, non-annotated words (with the exception of one tag -- ``, which is used for rare words such as uncommon proper nouns) for our model. This means that we just want to predict what the next words would be, not what they mean in context or their classes on a given sentence. the percentage of lung cancer deaths among the workers at the west `` mass. paper factory appears to be the highest for any asbestos workers studied in western industrialized countries he said  the plant which is owned by `` & `` co. was under contract with `` to make the cigarette filters  the finding probably will support those who argue that the u.s. should regulate the class of asbestos including `` more `` than the common kind of asbestos `` found in most schools and other buildings dr. `` said *Example of text from the dataset we are going to use, `ptb.train`*  Word EmbeddingsFor better processing, in this example, we will make use of [**word embeddings**]( [https://www.tensorflow.org/tutorials/word2vec/), which are **a way of representing sentence structures or words as n-dimensional vectors (where n is a reasonably high number, such as 200 or 500) of real numbers**. Basically, we will assign each word a randomly-initialized vector, and input those into the network to be processed. After a number of iterations, these vectors are expected to assume values that help the network to correctly predict what it needs to -- in our case, the probable next word in the sentence. This is shown to be very effective in Natural Language Processing tasks, and is a commonplace practice.$$Vec("Example") = [0.02, 0.00, 0.00, 0.92, 0.30,...]$$Word Embedding tends to group up similarly used words *reasonably* together in the vectorial space. For example, if we use T-SNE (a dimensional reduction visualization algorithm) to flatten the dimensions of our vectors into a 2-dimensional space and use the words these vectors represent as their labels, we might see something like this:*T-SNE Mockup with clusters marked for easier visualization*As you can see, words that are frequently used together, in place of each other, or in the same places as them tend to be grouped together -- being closer together the higher these correlations are. For example, "None" is pretty semantically close to "Zero", while a phrase that uses "Italy" can probably also fit "Germany" in it, with little damage to the sentence structure. A vectorial "closeness" for similar words like this is a great indicator of a well-built model.---  We need to import the necessary modules for our code. We need **`numpy` and `tensorflow`**, obviously. Additionally, we can import directly the **`tensorflow.models.rnn.rnn`** model, which includes the function for building RNNs, and **`tensorflow.models.rnn.ptb.reader`** which is the helper module for getting the input data from the dataset we just downloaded.If you want to learm more take a look at https://www.tensorflow.org/versions/r0.11/api_docs/python/rnn_cell/<jupyter_code>from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import time
import numpy as np
import tensorflow as tf
!wget -q -O /resources/data/ptb.zip https://ibm.box.com/shared/static/z2yvmhbskc45xd2a9a4kkn6hg4g4kj5r.zip
!unzip -o /resources/data/ptb.zip -d /resources/
!cp /resources/ptb/reader.py .
import reader<jupyter_output><empty_output><jupyter_text>Building the LSTM model for Language ModelingNow that we know exactly what we are doing, we can start building our model using TensorFlow. The very first thing we need to do is download and extract the `simple-examples` dataset, which can be done by executing the code cell below.<jupyter_code>!wget http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz 
!tar xzf simple-examples.tgz -C /resources/data/<jupyter_output><empty_output><jupyter_text>Additionally, for the sake of making it easy to play around with the model's hyperparameters, we can declare them beforehand. Feel free to change these -- you will see a difference in performance each time you change those!<jupyter_code>#Initial weight scale
init_scale = 0.1
#Initial learning rate
learning_rate = 1.0
#Maximum permissible norm for the gradient (For gradient clipping -- another measure against Exploding Gradients)
max_grad_norm = 5
#The number of layers in our model
num_layers = 2
#The total number of recurrence steps, also known as the number of layers when our RNN is "unfolded"
num_steps = 20
#The number of processing units (neurons) in the hidden layers
hidden_size = 200
#The maximum number of epochs trained with the initial learning rate
max_epoch = 4
#The total number of epochs in training
max_max_epoch = 13
#The probability for keeping data in the Dropout Layer (This is an optimization, but is outside our scope for this notebook!)
#At 1, we ignore the Dropout Layer wrapping.
keep_prob = 1
#The decay for the learning rate
decay = 0.5
#The size for each batch of data
batch_size = 30
#The size of our vocabulary
vocab_size = 10000
#Training flag to separate training from testing
is_training = 1
#Data directory for our dataset
data_dir = "/resources/data/simple-examples/data/"<jupyter_output><empty_output><jupyter_text>Some clarifications for LSTM architecture based on the argumants:Network structure:- In this network, the number of LSTM cells are 2. To give the model more expressive power, we can add multiple layers of LSTMs to process the data. The output of the first layer will become the input of the second and so on.- The recurrence steps is 20, that is, when our RNN is "Unfolded", the recurrence step is 20.   - the structure is like:      - 200 input units -> [200x200] Weight -> 200 Hidden units (first layer) -> [200x200] Weight matrix  -> 200 Hidden units (second layer) ->  [200] weight Matrix -> 200 unit outputHidden layer:- Each LSTM has 200 hidden units which is equivalant to the dimensianality of the embedding words and output. Input layer: - The network has 200 input units. - Suppose each word is represented by an embedding vector of dimensionality e=200. The input layer of each cell will have 200 linear units. These e=200 linear units are connected to each of the h=200 LSTM units in the hidden layer (assuming there is only one hidden layer, though our case has 2 layers).- The input shape is [batch_size, num_steps], that is [30x20]. It will turn into [30x20x200] after embedding, and then 20x[30x200] There is a lot to be done and a ton of information to process at the same time, so go over this code slowly. It may seem complex at first, but if you try to ally what you just learned about language modelling to the code you see, you should be able to understand it.This code is adapted from the [PTBModel](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/ptb/ptb_word_lm.py) example bundled with the TensorFlow source code. Train dataThe story starts from data: - Train data is a list of words, represented by numbers - N=929589 numbers, e.g. [9971, 9972, 9974, 9975,...]- We read data as mini-batch of size b=30. Assume the size of each sentence is 20 words (num_steps = 20). Then it will take int(N/b*h)+1=1548 iterations for the learner to go through all sentences once. So, the number of iterators is 1548- Each batch data is read from train dataset of size 600, and shape of [30x20] First we start an interactive session:<jupyter_code>session=tf.InteractiveSession()
# Reads the data and separates it into training data, validation data and testing data
raw_data = reader.ptb_raw_data(data_dir)
train_data, valid_data, test_data, _ = raw_data<jupyter_output><empty_output><jupyter_text>Lets just read one mini-batch now and feed our network:<jupyter_code>itera = reader.ptb_iterator(train_data, batch_size, num_steps)
first_touple=itera.next()
x=first_touple[0]
y=first_touple[1]
x.shape<jupyter_output><empty_output><jupyter_text>Lets look at 3 sentences of our input x:<jupyter_code>x[0:3]
size = hidden_size<jupyter_output><empty_output><jupyter_text>we define 2 place holders to feed them with mini-batchs, that is x and y:<jupyter_code>_input_data = tf.placeholder(tf.int32, [batch_size, num_steps]) #[30#20]
_targets = tf.placeholder(tf.int32, [batch_size, num_steps]) #[30#20]<jupyter_output><empty_output><jupyter_text>lets defin a dictionary, and use it later to feed the placeholders with our first mini-batch:<jupyter_code>feed_dict={_input_data:x, _targets:y}<jupyter_output><empty_output><jupyter_text>For example, we can use it to feed _input_data:<jupyter_code>session.run(_input_data,feed_dict)<jupyter_output><empty_output><jupyter_text>In this step, we create the stacked LSTM, which is a 2 layer LSTM network:<jupyter_code>lstm_cell = tf.contrib.rnn.BasicLSTMCell(hidden_size, forget_bias=0.0)
stacked_lstm = tf.contrib.rnn.MultiRNNCell([lstm_cell] * num_layers)<jupyter_output><empty_output><jupyter_text>Also, we initialize the states of the nework: _initial_stateFor each LCTM, there are 2 state matrics, c_state and m_state.  c_state and m_state represent "Memory State" and "Cell State". Each hidden layer, has a vector of size 30, which keeps the states. so, for 200 hidden units in each LSTM, we have a matrix of size [30x200]<jupyter_code>_initial_state = stacked_lstm.zero_state(batch_size, tf.float32)
_initial_state<jupyter_output><empty_output><jupyter_text>lets look at the states, though they are all zero for now:<jupyter_code>session.run(_initial_state,feed_dict)<jupyter_output><empty_output><jupyter_text>EmbeddingsWe create the embeddings for our input data. embedding is dictionary of [10000x200] for all 10000 unique words.<jupyter_code>embedding = tf.get_variable("embedding", [vocab_size, hidden_size])  #[10000x200]
session.run(tf.global_variables_initializer())
session.run(embedding, feed_dict)<jupyter_output><empty_output><jupyter_text>embedding_lookup goes to each row of input_data, and for each word in the row/sentence, finds the correspond vector in embedding.It creates a [30*20*200] matrix, so, the first elemnt of __inputs__ (the first sentence), is a matrix of 20x200, which each row of it is vector representing a word in the sentence.<jupyter_code># Define where to get the data for our embeddings from
inputs = tf.nn.embedding_lookup(embedding, _input_data)  #shape=(30, 20, 200) 
inputs
session.run(inputs[0], feed_dict)<jupyter_output><empty_output><jupyter_text>Constructing Recurrent Neural Networks__tf.nn.dynamicrnn()__ creates a recurrent neural network using __stacked_lstm__ which is an instance of RNNCell. The input should be a Tensor of shape: [batch_size, max_time, ...], in our case it would be (30, 20, 200)This method, returns a pair (outputs, new_state) where:- outputs is a length T list of outputs (one for each input), or a nested tuple of such elements.- new_state is the final state<jupyter_code>outputs, new_state =  tf.nn.dynamic_rnn(stacked_lstm, inputs, initial_state=_initial_state)<jupyter_output><empty_output><jupyter_text>so, lets look at the outputs. The output of the stackedLSTM comes from 200 hidden_layer, and in each time step(=20), one of them get activated. we use the linear activation to map the 200 hidden layer to a [?x10 matrix]<jupyter_code>outputs
session.run(tf.global_variables_initializer())
session.run(outputs[0], feed_dict)<jupyter_output><empty_output><jupyter_text>Lets reshape the output tensor from  [30 x 20 x 200] to [600 x 200]<jupyter_code>output = tf.reshape(outputs, [-1, size])
output
session.run(output[0], feed_dict)<jupyter_output><empty_output><jupyter_text>logistic unitNow, we create a logistic unit to return the probability of the output word. That is, mapping the 600Softmax = [600 x 200]* [200 x 1000]+ [1 x 1000] -> [600 x 1000]<jupyter_code>softmax_w = tf.get_variable("softmax_w", [size, vocab_size]) #[200x1000]
softmax_b = tf.get_variable("softmax_b", [vocab_size]) #[1x1000]
logits = tf.matmul(output, softmax_w) + softmax_b
session.run(tf.global_variables_initializer())
logi = session.run(logits, feed_dict)
logi.shape
First_word_output_probablity = logi[0]
First_word_output_probablity.shape<jupyter_output><empty_output><jupyter_text>PredictionThe maximum probablity<jupyter_code>embedding_array= session.run(embedding, feed_dict)
np.argmax(First_word_output_probablity)<jupyter_output><empty_output><jupyter_text>So, what is the ground truth for the first word of first sentence?<jupyter_code>y[0][0]<jupyter_output><empty_output><jupyter_text>Also, you can get it from target tensor, if you want to find the embedding vector:<jupyter_code>_targets<jupyter_output><empty_output><jupyter_text>It is time to compare logit with target<jupyter_code>targ = session.run(tf.reshape(_targets, [-1]), feed_dict) 
first_word_target_code= targ[0]
first_word_target_code
first_word_target_vec = session.run( tf.nn.embedding_lookup(embedding, targ[0]))
first_word_target_vec<jupyter_output><empty_output><jupyter_text>Objective functionNow we want to define our objective function. Our objective is to minimize loss function, that is, to minimize the average negative log probability of the target words:loss=−1N∑i=1Nln⁡ptargeti  This function is already implimented and available in TensorFlow through __sequence_loss_by_example__ so we can just use it here. __sequence_loss_by_example__ is weighted cross-entropy loss for a sequence of logits (per example).  Its arguments:  logits: List of 2D Tensors of shape [batch_size x num_decoder_symbols].  targets: List of 1D batch-sized int32 Tensors of the same length as logits.  weights: List of 1D batch-sized float-Tensors of the same length as logits.<jupyter_code>loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example([logits], [tf.reshape(_targets, [-1])],[tf.ones([batch_size * num_steps])])<jupyter_output><empty_output><jupyter_text>loss is a 1D batch-sized float Tensor [600x1]: The log-perplexity for each sequence.<jupyter_code>session.run(loss, feed_dict)
cost = tf.reduce_sum(loss) / batch_size

session.run(tf.global_variables_initializer())
session.run(cost, feed_dict)<jupyter_output><empty_output><jupyter_text>Now, lets store the new state as final state<jupyter_code># 
final_state = new_state<jupyter_output><empty_output><jupyter_text>TrainingTo do gradient clipping in TensorFlow we have to take the following steps:1. Define the optimizer.2. Extract variables that are trainable.3. Calculate the gradients based on the loss function.4. Apply the optimizer to the variables / gradients tuple.  1. Define Optimizer__GradientDescentOptimizer__ constructs a new gradient descent optimizer. Later, we use constructed __optimizer__ to compute gradients for a loss and apply gradients to variables.<jupyter_code># Create a variable for the learning rate
lr = tf.Variable(0.0, trainable=False)
# Create the gradient descent optimizer with our learning rate
optimizer = tf.train.GradientDescentOptimizer(lr)<jupyter_output><empty_output><jupyter_text>2. Trainable Variables Definining a variable, if you passed _trainable=True_, the _Variable()_ constructor automatically adds new variables to the graph collection __GraphKeys.TRAINABLE_VARIABLES__. Now, using _tf.trainable_variables()_ you can get all variables created with __trainable=True__.<jupyter_code># Get all TensorFlow variables marked as "trainable" (i.e. all of them except _lr, which we just created)
tvars = tf.trainable_variables()
tvars<jupyter_output><empty_output><jupyter_text>we can find the name and scope of all variables:<jupyter_code>tvars=tvars[3:]
[v.name for v in tvars]<jupyter_output><empty_output><jupyter_text>3. Calculate the gradients based on the loss function<jupyter_code>cost
tvars<jupyter_output><empty_output><jupyter_text>Gradient:The gradient of a function (line) is the slope of the line, or the rate of change of a function. It's a vector (a direction to move) that points in the direction of greatest increase of the function, and calculated by __derivative__ operation.First lets recall the gradient function using an toy example:$$ z=\left(2x^2+3xy\right)$$<jupyter_code>var_x = tf.placeholder(tf.float32)
var_y = tf.placeholder(tf.float32) 
func_test = 2.0*var_x*var_x + 3.0*var_x*var_y
session.run(tf.global_variables_initializer())
feed={var_x:1.0,var_y:2.0}
session.run(func_test, feed)<jupyter_output><empty_output><jupyter_text>The __tf.gradients()__ function allows you to compute the symbolic gradient of one tensor with respect to one or more other tensors—including variables. __tf.gradients(func,xs)__ constructs symbolic partial derivatives of sum of __func__ w.r.t. _x_ in __xs__. Now, lets look at the derivitive w.r.t. __var_x__:$$ \frac{\partial \:}{\partial \:x}\left(2x^2+3xy\right)=4x+3y $$<jupyter_code>var_grad = tf.gradients(func_test, [var_x])
session.run(var_grad,feed)<jupyter_output><empty_output><jupyter_text>the derivitive w.r.t. __var_y__:$$ \frac{\partial \:}{\partial \:x}\left(2x^2+3xy\right)=3x $$<jupyter_code>var_grad = tf.gradients(func_test, [var_y])
session.run(var_grad,feed)<jupyter_output><empty_output><jupyter_text>Now, we can look at gradients w.r.t all variables:<jupyter_code>tf.gradients(cost, tvars)
grad_t_list = tf.gradients(cost, tvars)
#sess.run(grad_t_list,feed_dict)<jupyter_output><empty_output><jupyter_text>now, we have a list of tensors, t-list. We can use it to find clipped tensors. __clip_by_global_norm__ clips values of multiple tensors by the ratio of the sum of their norms.__clip_by_global_norm__ get _t-list_ as input and returns 2 things: - a list of clipped tensors, so called _list_clipped_  - the global norm (global_norm) of all tensors in t_list<jupyter_code>max_grad_norm
# Define the gradient clipping threshold
grads, _ = tf.clip_by_global_norm(grad_t_list, max_grad_norm)
grads
session.run(grads,feed_dict)<jupyter_output><empty_output><jupyter_text>4. Apply the optimizer to the variables / gradients tuple.<jupyter_code># Create the training TensorFlow Operation through our optimizer
train_op = optimizer.apply_gradients(zip(grads, tvars))
session.run(tf.global_variables_initializer())
session.run(train_op,feed_dict)<jupyter_output><empty_output><jupyter_text>We learned how the model is build step by step. Noe, let's then create a Class that represents our model. This class needs a few things:- We have to create the model in accordance with our defined hyperparameters- We have to create the placeholders for our input data and expected outputs (the real data)- We have to create the LSTM cell structure and connect them with our RNN structure- We have to create the word embeddings and point them to the input data- We have to create the input structure for our RNN- We have to instanciate our RNN model and retrieve the variable in which we should expect our outputs to appear- We need to create a logistic structure to return the probability of our words- We need to create the loss and cost functions for our optimizer to work, and then create the optimizer- And finally, we need to create a training operation that can be run to actually train our model<jupyter_code>class PTBModel(object):

    def __init__(self, is_training):
        ######################################
        # Setting parameters for ease of use #
        ######################################
        self.batch_size = batch_size
        self.num_steps = num_steps
        size = hidden_size
        self.vocab_size = vocab_size
        
        ###############################################################################
        # Creating placeholders for our input data and expected outputs (target data) #
        ###############################################################################
        self._input_data = tf.placeholder(tf.int32, [batch_size, num_steps]) #[30#20]
        self._targets = tf.placeholder(tf.int32, [batch_size, num_steps]) #[30#20]

        ##########################################################################
        # Creating the LSTM cell structure and connect it with the RNN structure #
        ##########################################################################
        # Create the LSTM unit. 
        # This creates only the structure for the LSTM and has to be associated with a RNN unit still.
        # The argument n_hidden(size=200) of BasicLSTMCell is size of hidden layer, that is, the number of hidden units of the LSTM (inside A).
        # Size is the same as the size of our hidden layer, and no bias is added to the Forget Gate. 
        # LSTM cell processes one word at a time and computes probabilities of the possible continuations of the sentence.
        lstm_cell = tf.contrib.rnn.BasicLSTMCell(size, forget_bias=0.0)
        
        # Unless you changed keep_prob, this won't actually execute -- this is a dropout wrapper for our LSTM unit
        # This is an optimization of the LSTM output, but is not needed at all
        if is_training and keep_prob < 1:
            lstm_cell = tf.contrib.rnn.DropoutWrapper(lstm_cell, output_keep_prob=keep_prob)
        
        # By taking in the LSTM cells as parameters, the MultiRNNCell function junctions the LSTM units to the RNN units.
        # RNN cell composed sequentially of multiple simple cells.
        stacked_lstm = tf.contrib.rnn.MultiRNNCell([lstm_cell] * num_layers)

        # Define the initial state, i.e., the model state for the very first data point
        # It initialize the state of the LSTM memory. The memory state of the network is initialized with a vector of zeros and gets updated after reading each word.
        self._initial_state = stacked_lstm.zero_state(batch_size, tf.float32)

        ####################################################################
        # Creating the word embeddings and pointing them to the input data #
        ####################################################################
        with tf.device("/cpu:0"):
            # Create the embeddings for our input data. Size is hidden size.
            embedding = tf.get_variable("embedding", [vocab_size, size])  #[10000x200]
            # Define where to get the data for our embeddings from
            inputs = tf.nn.embedding_lookup(embedding, self._input_data)

        # Unless you changed keep_prob, this won't actually execute -- this is a dropout addition for our inputs
        # This is an optimization of the input processing and is not needed at all
        if is_training and keep_prob < 1:
            inputs = tf.nn.dropout(inputs, keep_prob)

        ############################################
        # Creating the input structure for our RNN #
        ############################################
        # Input structure is 20x[30x200]
        # Considering each word is represended by a 200 dimentional vector, and we have 30 batchs, we create 30 word-vectors of size [30xx2000]
        #inputs = [tf.squeeze(input_, [1]) for input_ in tf.split(1, num_steps, inputs)]
        # The input structure is fed from the embeddings, which are filled in by the input data
        # Feeding a batch of b sentences to a RNN:
        # In step 1,  first word of each of the b sentences (in a batch) is input in parallel.  
        # In step 2,  second word of each of the b sentences is input in parallel. 
        # The parallelism is only for efficiency.  
        # Each sentence in a batch is handled in parallel, but the network sees one word of a sentence at a time and does the computations accordingly. 
        # All the computations involving the words of all sentences in a batch at a given time step are done in parallel. 

        ####################################################################################################
        # Instanciating our RNN model and retrieving the structure for returning the outputs and the state #
        ####################################################################################################
        
        outputs, state = tf.nn.dynamic_rnn(stacked_lstm, inputs, initial_state=self._initial_state)

        #########################################################################
        # Creating a logistic unit to return the probability of the output word #
        #########################################################################
        output = tf.reshape(outputs, [-1, size])
        softmax_w = tf.get_variable("softmax_w", [size, vocab_size]) #[200x1000]
        softmax_b = tf.get_variable("softmax_b", [vocab_size]) #[1x1000]
        logits = tf.matmul(output, softmax_w) + softmax_b

        #########################################################################
        # Defining the loss and cost functions for the model's learning to work #
        #########################################################################
        loss = tf.contrib.legacy_seq2seq.sequence_loss_by_example([logits], [tf.reshape(self._targets, [-1])],
                                                      [tf.ones([batch_size * num_steps])])
        self._cost = cost = tf.reduce_sum(loss) / batch_size

        # Store the final state
        self._final_state = state

        #Everything after this point is relevant only for training
        if not is_training:
            return

        #################################################
        # Creating the Training Operation for our Model #
        #################################################
        # Create a variable for the learning rate
        self._lr = tf.Variable(0.0, trainable=False)
        # Get all TensorFlow variables marked as "trainable" (i.e. all of them except _lr, which we just created)
        tvars = tf.trainable_variables()
        # Define the gradient clipping threshold
        grads, _ = tf.clip_by_global_norm(tf.gradients(cost, tvars), max_grad_norm)
        # Create the gradient descent optimizer with our learning rate
        optimizer = tf.train.GradientDescentOptimizer(self.lr)
        # Create the training TensorFlow Operation through our optimizer
        self._train_op = optimizer.apply_gradients(zip(grads, tvars))

    # Helper functions for our LSTM RNN class

    # Assign the learning rate for this model
    def assign_lr(self, session, lr_value):
        session.run(tf.assign(self.lr, lr_value))

    # Returns the input data for this model at a point in time
    @property
    def input_data(self):
        return self._input_data

    # Returns the targets for this model at a point in time
    @property
    def targets(self):
        return self._targets

    # Returns the initial state for this model
    @property
    def initial_state(self):
        return self._initial_state

    # Returns the defined Cost
    @property
    def cost(self):
        return self._cost

    # Returns the final state for this model
    @property
    def final_state(self):
        return self._final_state

    # Returns the current learning rate for this model
    @property
    def lr(self):
        return self._lr

    # Returns the training operation defined for this model
    @property
    def train_op(self):
        return self._train_op<jupyter_output><empty_output><jupyter_text>With that, the actual structure of our Recurrent Neural Network with Long Short-Term Memory is finished. What remains for us to do is to actually create the methods to run through time -- that is, the `run_epoch` method to be run at each epoch and a `main` script which ties all of this together.What our `run_epoch` method should do is take our input data and feed it to the relevant operations. This will return at the very least the current result for the cost function.<jupyter_code>##########################################################################################################################
# run_epoch takes as parameters the current session, the model instance, the data to be fed, and the operation to be run #
##########################################################################################################################
def run_epoch(session, m, data, eval_op, verbose=False):

    #Define the epoch size based on the length of the data, batch size and the number of steps
    epoch_size = ((len(data) // m.batch_size) - 1) // m.num_steps
    start_time = time.time()
    costs = 0.0
    iters = 0
    #state = m.initial_state.eval()
    #m.initial_state = tf.convert_to_tensor(m.initial_state) 
    #state = m.initial_state.eval()
    state = session.run(m.initial_state)
    
    #For each step and data point
    for step, (x, y) in enumerate(reader.ptb_iterator(data, m.batch_size, m.num_steps)):
 
        #Evaluate and return cost, state by running cost, final_state and the function passed as parameter
        cost, state, _ = session.run([m.cost, m.final_state, eval_op],
                                     {m.input_data: x,
                                      m.targets: y,
                                      m.initial_state: state})
        
        #Add returned cost to costs (which keeps track of the total costs for this epoch)
        costs += cost
        
        #Add number of steps to iteration counter
        iters += m.num_steps

        if verbose and step % (epoch_size // 10) == 10:
            print("%.3f perplexity: %.3f speed: %.0f wps" % (step * 1.0 / epoch_size, np.exp(costs / iters),
              iters * m.batch_size / (time.time() - start_time)))

    # Returns the Perplexity rating for us to keep track of how the model is evolving
    return np.exp(costs / iters)<jupyter_output><empty_output><jupyter_text>Now, we create the `main` method to tie everything together. The code here reads the data from the directory, using the `reader` helper module, and then trains and evaluates the model on both a testing and a validating subset of data.<jupyter_code># Reads the data and separates it into training data, validation data and testing data
raw_data = reader.ptb_raw_data(data_dir)
train_data, valid_data, test_data, _ = raw_data
#Initializes the Execution Graph and the Session
with tf.Graph().as_default(), tf.Session() as session:
    initializer = tf.random_uniform_initializer(-init_scale,init_scale)
    
    # Instantiates the model for training
    # tf.variable_scope add a prefix to the variables created with tf.get_variable
    with tf.variable_scope("model", reuse=None, initializer=initializer):
        m = PTBModel(is_training=True)
        
    # Reuses the trained parameters for the validation and testing models
    # They are different instances but use the same variables for weights and biases, they just don't change when data is input
    with tf.variable_scope("model", reuse=True, initializer=initializer):
        mvalid = PTBModel(is_training=False)
        mtest = PTBModel(is_training=False)

    #Initialize all variables
    tf.global_variables_initializer().run()

    for i in range(max_max_epoch):
        # Define the decay for this epoch
        lr_decay = decay ** max(i - max_epoch, 0.0)
        
        # Set the decayed learning rate as the learning rate for this epoch
        m.assign_lr(session, learning_rate * lr_decay)

        print("Epoch %d : Learning rate: %.3f" % (i + 1, session.run(m.lr)))
        
        # Run the loop for this epoch in the training model
        train_perplexity = run_epoch(session, m, train_data, m.train_op,
                                   verbose=True)
        print("Epoch %d : Train Perplexity: %.3f" % (i + 1, train_perplexity))
        
        # Run the loop for this epoch in the validation model
        valid_perplexity = run_epoch(session, mvalid, valid_data, tf.no_op())
        print("Epoch %d : Valid Perplexity: %.3f" % (i + 1, valid_perplexity))
    
    # Run the loop in the testing model to see how effective was our training
    test_perplexity = run_epoch(session, mtest, test_data, tf.no_op())
    
    print("Test Perplexity: %.3f" % test_perplexity)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10769703058718463, 'fraction_numerical': 0.013172583165885243, 'mean_word_length': 4.381138309055414, 'pattern_counts': {'":': 0, '<': 155, '<?xml version=': 0, '>': 160, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '279866', 'license': 'MIT', 'n_tokens_mistral': 10315, 'n_tokens_neox': 9705, 'n_words': 4172, 'path': 'cognitive class - deep learning tensorflow/ML0120EN-3.3-Review-LSTM-LanguageModelling.ipynb', 'repo_name': 'fotavio16/PycharmProjects', 'size': '35832'}