Understanding and Implementing Recurrent Neural Networks using Python
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Sachdeva, Anmol Krishan
Recurrent Neural Networks (RNNs) have become famous over time due to their property of retaining internal memory. These neural nets are widely used in recognizing patterns in sequences of data, like numerical timer series data, images, handwritten text, spoken words, genome sequences, and much more. Since these nets possess memory, there is a certain analogy that we can make to the human brain in order to learn how RNNs work. RNNs can be thought of as a network of neurons with feedback connections, unlike feedforward connections which exist in other types of Artificial Neural Networks. The flow of talk will be as follows: - Self Introduction - Introduction to Deep Learning - Artificial Neural Networks (ANNs) - Diving DEEP into Recurrent Neural Networks (RNNs) - Comparing Feedforward Networks with Feedback Networks - Quick walkthrough: Implementing RNNs using Python (Keras) - Understanding Backpropagation Through Time (BPTT) and Vanishing Gradient Problem - Towards more sophisticated RNNs: Gated Recurrent Units (GRUs)/Long Short-Term Memory (LSTMs) - End of talk - Questions and Answers Session
 All right. A very warm welcome and good morning all of you. My name is Anmol Krishnadevar and the topic for today is understanding and implementing recurrent neural networks. So before starting let's first see an example. If we are reading a book and it has say five chapters, we are on chapter one, we have read that, we are going to chapter two, we are starting reading it, but suddenly we forgot everything that was there in chapter one. So how will we be able to understand chapter two? It means short-term memory plays a very crucial role. On that motion I would like to start with the recurrent neural networks. So Martin has already introduced me. I'll be skipping this. For requisites you should be aware of Python language. You should have a decent knowledge of artificial neural networks and elementary linear algebra. So recurrent neural networks can be thought of as the neural networks which persist information and we can think it off as sequential processes, basically influencing decisions. So in contrast to the traditional neural networks which don't persist information, recurrent neural networks are having this advantage over those. So we can think recurrent neural networks as the networks having loops to themselves. I'll be explaining the architecture now. And these are one of the most complex supervised deep learning algorithms we have today. So consider a normal neural network and it has many layers. X naught, X1, X2 is basically the input that we are providing to the layer. A is a hidden layer. So the input from X naught goes to A and the input gets transferred from one hidden layer to another hidden layer. Simultaneously it gets transferred from the other hidden layer to the next hidden layer and we can think it off squashing this thing that you see on your right hand side into the thing that is known as the vanilla RNN. That means it is having loop to itself. RNNs can be of many architectures. The first one is one to many architecture in which you provide one input and you can get many outputs. So you can think of, say, you provide with an image and it provides with certain caption. So an image is mapped into caption of, say, 506 words. So it is one image getting mapped into 506 words that is one to many transformation of RNN. Likewise we have many to one. So many to one can be thought of as, say, we are having video frames and we are generating text out of the video. So it can be mapped to many to many or many to one. So we can get from a video a sentence and similarly we can have many to many transformation also. So recurrent neural networks have the following applications that is image captioning, subtitle generation, time series classification, language modeling, natural language processing, even JAPO development is also based on RNNs nowadays. So there is a major problem in the vanilla RNN. That is if the RNN is having so many layers, so adjusting the weights at each hidden layer is a problem. That means once the information from X naught is transferred to A, it is multiplied by some weight matrix, then transferred to the next layer, then transferred to next, then transferred to next. And simultaneously the output gets generated on the top. Now we have something called as loss function. That is the difference between the actual output and the predicted output. So if the loss functions for say ht plus one generate some error, so there's some like error of say 0.15 percent and it's need to be back propagated throughout the network and back propagating through a network that is so large is difficult because each time you can think of say W is getting multiplied at each layer and multiplying W that is between zero and one, multiplying anything by zero and one, between zero and one say 0.2. If we multiply something by zero point two multiple times then it will be tending to very small value. So it will be like the gradient will be propagating through the whole chain but it will be taking much time to train. So it is not feasible solution. So vanilla RNN suffers from the vanishing gradient problem. Likewise we have the exploding gradient problem. So exploding gradient problem is when the value of W is greater than one. So if you multiply simultaneously a number by say a factor of more than one then it will tend to always go and multiply to many huge values. Now the vanishing gradient problem can be thought of as W is less than one and exploding gradient can be thought of as W is greater than one. To solve these vanishing gradient problem and the exploding gradient problems we use certain techniques. So for exploding gradient problem we have truncated back propagation. So in truncated back propagation we divide the whole set into certain batches and in those certain batches we just back propagate within those certain batches sequentially. In system of rewards and penalties it is like reinforcement learning. So we provide rewards for say if the back propagation is doing well else we provide penalties. And we have gradient clipping. If the gradient goes beyond some range then we just clip that gradient and we don't propagate it through the network. For vanishing gradient problem we have smart weight initialization that is like a guesswork. Then we have equestrate networks and we have LSTM. So I'll be talking today of LSTM that is long short-term memory and it is one of the most used variants of RNN. So LSTM is one of the most used variants of RNN and the approach for LSTM is making the weight W equal to one. So you can think of you are not having W less than one, you are not having W greater than one so what else we can do? We can just make W equal to one. Now coming to the architecture of the LSTM C is the new introduction in here. HT is the state of the current input the hidden layer and CT can be thought of as a cell state. So in here if we provide the input XT and the hidden layer and we transfer this thing to four states FIG O. F is the you can say the final gate of forget gate, I is the input gate, G is another gate, there's no name for it and O is the output gate. We take we apply the function like the element wise multiplication and the addition using this formula. So CT is basically F times that is forget gate times the CT minus one that is the previous state of the cell that was there. It is basically forgetting some part of the memory. So it is like you were reading something and you actually forgot some part of it but you retain certain part of it. So this is actually forgetting major part of it and retaining some part of it and that some part is used for training the rest of the network and this is actually applied through the tanh function and the output gate is multiplied and we get the output of the hidden layer. So comparing the RNN and the LSTM, it is we can say the LSTM has a sophisticated architecture in terms that it has a cell state and using this cell state it is like a super highway through which W is not getting changed. So we are actually changing the element wise fraction that is F and we are not changing the weight matrix. So multiplying anything by element or scalar multiplication is much simpler than the matrix multiplication which is involved in the weight matrix multiplication that is used in the vanilla RNN. So that is why the vanishing gradient problem goes off in LSTM. Now we can start building the LSTM and the code I'll be providing for that. So you can just join in this, you can just implement this as it is now. I'll be just shifting to the implementation thing. So the major task of implementing the RNN involves the data preprocessing. Data preprocessing then building the recurrent neural model and making the predictions and visualizations. So before implementing we need certain libraries that is Keras library then we use scikit-learn, we use tensorflow and yeah. So first task is data preprocessing. So we import the NumPy library, we import the Matplot library, we import the Pandas library. The NumPy library is for visualizing the results, is for manipulation Matplot is for visualizing the results and Pandas is for managing the data sets because the Keras library, yeah, zoom. Okay. Can you see now? Okay. So since the Keras library doesn't support the data frames of Pandas, we need to use the NumPy arrays for Keras library. Now before starting, let me introduce you to the problem. So today we'll be predicting the stock price for the Google for first month of 2017. So these are the records that you can see. These are the records that you can see. So these are 20 records that is each month has 20 financial days, that means weekdays. So this is the record that we have to predict on and the training data set is this. So we'll be predicting the open stock prices of Google for first month of 2017 and we'll be training on the year 2012 to 2016, so five years of data. Now actually the prediction should look like this. So if I take these two columns and just insert the plot for it. So the result that we should predict should be like this. So today our task is to mimic this behavior of the stock price in January 2017. Now let's get started So the training data set is I'm calling it Google stock price train.csv. I'm importing it as a data set, the Pandas data frame. So I'm using pd.readcsv. Then since this data frame contains many columns but we only require the open stock prices, the open stock prices that is column one. So I use the iLock function for selecting, the first parameter in iLock is the column that means it will be selecting all the rows and the column, the second parameter is the column parameter that is one to two, one points to the open and two basically points to the next column but since the second value is omitted so it is just considering the first column that is the open column and dot value is for converting it to the NumPy arrays. So we are just converting the Pandas data frame to the NumPy array. Now we need to scale this data and we'll be using the scikit-learn preprocessing library and from that we'll be using the minmax scalar class. So sc is the object of minmax scalar class and the feature range parameter here tells that you need to convert the values of the column between the range 0 to 1 so everything will be scaled between 0 to 1 and after that we just fit and transform. So fit and transform means fitting basically means normalizing the data so it actually takes the minimum value from that column and the maximum value from that column and it puts it, the transform puts it into the normalization function. The normalization function can be the standardized function or the normal normalization function so in standardized function we take the mean value and we subtract it from the actual value and then we divide it by mean value but in normalization function we take the min value we subtract it from the main value and we divide it by the difference between the max value and the min value. So it is a basic normalization thing that we are doing so fit and transform does this thing. After that we need to create the training data, the training list and the predicted ground truth list. So X train is a list, an empty list and Y train is an empty list that is for the training you can say set and the ground truth set. So we have 1 to 5-8 records here in the stock prediction data so we have 1 to 5-8 records these are 1 to 5-8 records and we are taking the time step as 60 so what is time step? Basically time step is a very important concept here in the recurrent neural networks. It is basically how many observations you will be taking into consideration for training the next value or you can say if you are focusing on just predicting the ith value then what set of values you will be taking into consideration before that ith value. So it is basically I'm starting from the 60th record it means it will be taking the data of three months training on that and predicting the 61st value. So it is just an assumption you can take 20 days, 18 days, anything but it is like during my testing I took 60 and it produced great results so I'm using 60 as the time step value and 1 to 5-8 is the number of records. So I append to the empty list the training scaled values from the I-60th value to the current value so that is basically appending it is making an array in which one record contains 60 previous values from the current value so it is like a 6 you can say n cross 60 matrix that is getting created. Now the wire train is basically the ground truth and the ground truth will be just having the current value so it is from I to I plus 1 and 0 is basically the column that is the first column because we have already cleaned our data and we are just having one column for open stock prices so 0 points that thing. Now X train comma Y train equals NP dot array so it is converting to numpy array so we are generating X train numpy array and Y train numpy array then we need to reshape the X train array so reshaping means that to work seamlessly with the you can say numpy arrays and the RN and stuff we need to add an extra dimension to the this existing numpy array so basically this is this basically X train dot shape is the number of rows X train dot shape one is number of columns and one is basically the indicator column that is the open stock column so we are just reshaping to build it a 3d array from a 2d array. Now the first step of data pre-processing has been done and we are on to the next step of building the RNN does anyone have any doubt in this yeah yeah so basically we need to train the data we need to say we are not having we are having values say 700 800 1000 to say 20 20 to 5 in the stock prices and it is not ideal it will take a lot of time and it will not generate good results if the data is not scaled between some value say 0 or 1 so I have provided the RN 0 to 1 to make it more cleaner so it will be approximated to the nearest value so it is just cleaning the data and smoothening the data so scaling has that use yeah yeah mm-hmm right so basically the question that you are asking is a very good question and we use something called grid search for this that is basically a part of hyper parameter tuning so these are basically pointing towards the hyper parameter stuff so 60 is like hyper parameter only which is making or which is amending how the predictions are being made so what we can do is after this is done like this is the first school for you all people but I tried it with certain parameters and in the hyper parameter tuning the grid search library provides you with you can say you can provide arrays of values say I have something called say time step so I provide time step equal to 30 comma 60 comma 300 suppose and I have another value say optimizer and I take three optimizers suppose which are used for this type of regression this is a regression problem because we are dealing with the continuous values so we take say rms prop optimizer that I'll be telling you more about and we take atom optimizer and we say n atom optimizer so it will be doing a cross product so it will be take this parameter 30 time step cross it with the the first rms prop and generate result then 30 add them generate result 30 and atom generate result then it will be doing same thing with 60 rms prop 60 add them 60 and add them so it will be generating nine results and from those nine results you can actually get the matrix formed which will be showing which one is generating good accuracy and less loss so so using that grid search thing you can actually judge which hyper parameter to tune to what value so that is part of grid search thanks no I have not because it is like I don't have that much time for showing yeah so now building the RNN stuff so we are importing four classes sequential dense lstm and dropout sequential class is for basically inputting the first you can say set of inputs to the neural network dense class is for the last layer of the neural network that's the output layer lstm class is for making the hidden layers of the neural network and dropout is for something called as dropout regularization which I'll be telling you more about further so we are making a regressor since it's not a classification problem it's we are dealing with the continuous data set so it's a regression problem so we make an object of sequential class called regressor and we add the first layer to it so it's regressor.add so regressor.add will add layers to the neural network so first layer is the lstm layer the hidden layer that we are adding and units equal to 50 means the number of neurons you can say that we are introducing to the network so this layer will have 50 neurons and you can take it any value so it was again the hyper parameter stuff so I took 50 then return sequences equal to true means that the output of this layer will be getting forwarded to the next layer and input shape equals x-ray in dot shape one that points to the columns of the data set and one is for reshaping it to the third dimension so we are just using the the column and the third dimensional value for it we don't need to use the first row dimension value for it so this will be adding the first layer and adding the first layer may introduce overfitting say so to deal with the overfitting problem that is to say avoid the noise that is being created by the overfitting stuff we use the dropout regularization stuff so dropout actually means dropping out certain number of neurons from the layer so out of 50 I have chosen 0.2 that's 20 percent to be dropped out so it means it will be just considering 40 neurons for the next layer and out of those 50 neurons 10 neurons at random or say 20 percent of those 50 neurons will be dropped out at random so this actually avoids the overfitting problem yeah so basically it's a stacked del S T M so I am using four hidden layers in this and for the four layers I'm adding four dropouts so yeah mm-hmm yeah no no it's like during the epochs epochs is basically how many time the data the full data will be propagated forward and backward to lessen the loss the loss generated by you can say the gradient so it's actually for the number of epochs the dropout will be done at random so dropping out means just not considering 20 percent of those neurons randomly in one epoch and in the next epoch it will again be choosing random 20 percent and then be not considering so it's like a random stuff going on I chosen 100 epochs so it's fully propagating the data forward and backward and it training is done 100 times on this data so it's like for each epoch it is generating some random value some it's choosing some random neuron and it's dropping out 10 random neurons so it actually avoids the overfitting problem in that way yeah and then similarly I'm adding three more layers so this is regressor.add LSTM now you have added the first layer initially so you don't need to provide the input shape because LSTM class already knows that what is the input so from the second layer you don't need to provide this parameter input shape it will automatically take from the previous thing and this is the second layer then this is the third layer and the fourth layer in fourth layer we don't actually need the output return sequence because we need to pass it through the dense layer and not another LSTM layer so we'll be omitting the return sequence equal to true parameter we can state here return sequence equal to false but the default value of return sequence is false so we are not putting here the parameter and dense is the final output layer so basically it is the unit is equal to one means it will be having just one neuron that's the resultant neuron so dense is actually corresponding to the result of the neural network now the building of RNN has been done but the compilation phase is still lift compilation actually means compiling with a certain loss function and choosing the right optimizer so as i told you earlier rms prop and adam are the two optimizers that are used with RNN and adam is the one that usually works apart from say rn also it works with cnn also so it is a much wider optimizer that you can use and it gives good results again a hyper parameter tuning thing so i have chosen the optimizer as adam and since it is the regression stuff so we'll be using mean squared error loss function if it would have been say the classification stuff then we would have used the cross entropy or binary cross entropy loss function then we have compiled this and then we are ready to fit it so fitting the RNN for the training set so we pass the fit function x-train y-train epochs equal to 100 batch sizes 32 so it will be taking batch of 32 it will be dividing the data set into 32 say element batches and it will be working on those batches it's again hyper parameter tuning stuff so what i mean by hyper parameter tuning is that tuning the parameters of the classes in such a way that it provides good result or good predictions now the task tree is the prediction task so we have done the RNN building so we have done the data preprocessing stuff now is the task to predict the results so actually the data set for predicting was separate than the data set for testing so we just take the test data set and we merge that test data set with the actual prediction data set to make it a total data set which will contain every value so here we are just converting the read csv we are creating a data frame of the test data set and then we are again converting it to the numpy array as we did it with the price the prediction price stuff then we concatenate the open column of the prediction data set with the test column so it will be 1, 2, 5, 8 plus 20 records that's 1, 2, 7, 8 records in total because during the time of testing we need to have we need to consider the previous 60 records and for considering those previous 60 records we should have a data set incomplete so i'm just using this complete data set and x is zero is for vertical join input equals data set merged so basically it is taking it is creating another you can say numpy array which is based on the merge data set and it will be taking the values 60 values from the current state so counting from 1, 2, 5, 8 record number 1, 2, 5, 8 it will be taking into account the previous 60 values to predict the results of the record 1, 2, 5, 8 now we are reshaping it again and since we have already fitted the data earlier we'll just be transforming it so we are just taking the input numpy array and we are transforming the input numpy array for making the predictions and we use another list called x test and we append to this list the values from the last 20 rows that we need to test and then we form the numpy array of this and we just shape it again reshape it again now the prediction can be done using the predict function of the regressor class the sequence the class that we formed so regressor.predict basically makes prediction on the x test list that we just formed the numpy array and inverse transform is the function that is used to inverse the normalization stuff that we did so now we are just you can say we are inversing the scaling that we did we made the scaling from 0 to 1 we normalized the results from 0 to 1 we are just inversing the transform to actually convert them into the real value so it will be something like 0 point something is converted into 770 that is say a stock price for one record and then we plot these results so since the time is short i have already run this thing and paacham so it was say 100 epochs and you can see that i'll just show you say you see that with increasing number of epochs the loss function is decreasing so the value of loss is decreasing so the loss must have started with say some value 0.3 or something 0.0034 and it's now at the end of 100 epochs it's 0.001315 so you can see as i'm going up the loss function is increasing so the value of loss function is increasing it means that the training has been done correctly and we have achieved certain level of accuracy so that the loss function has minimized its value from say it's in 30s or 40s say 0.0030 to 0.13 and we can see that the plot that we have is following the trend of ups and downs of the market but it's not actually predicting the correct values though it is following the trend that's like if the value of the stock price is going up it is following that trend so blue is the predicted one and red is the actual one so using these parameters i was able to generate this result doing hyper parameter tuning with grid search will definitely improvise these results and it will actually match with some error with the actual red line so it is like just following the trend with the red thing so we can see that it is actually following following the trend so we can see on x-axis at point 2.5 it is just going down then it's following the upward trend and then it's being stable at the end so it was this prediction that using these hyper parameters i was able to make so the source code for this i'll be updating my slides but it's on it's on github it's at this address i'll be just uploading my slides and it will get uploaded on the session page and after that so these are my acknowledgements professor martin christian then my supervisors christopher wula for creating good blogs and yeah if you have any questions then you can ask yeah thanks thank you very much we have time for a couple of questions thank you i wanted to ask how did you tune your architecture why four layers okay so when we deal with RNNs we cannot go beyond say we generally choose two three or four layers because it actually doesn't make any sense going beyond four layers as the results will not improve over choosing four layers so for just showing the implementation stuff i used four layers and it was just making some good assumptions when compared with using three layers or two layers but generally in RNNs we just don't go beyond four layers it can be the case that if we are using RNNs with some other neural network say CNN for say image captioning task then it will be another scenario we'll be getting the results of the vector generated from the CNN and we'll be transferring those CNN results into the first hidden layer that's the H0 thing and those results will be just making further you can say vectors forming vectors which will then be propagated throughout the chain so it will be having more impact on the RNN training and in that case we can just lower down from four layers to two layers because it is already having lots of information from the CNN stuff so it is basically based on that thing so if we consider a basic example that's the perfect roommate example that is very famous in field of RNN if you have a roommate who cooks food for you and he cooks say apple pie chicken and say anything xyz anything so on sunny day he cooks say apple pie and if it's another sunny day then he just goes off the duty and doesn't cook anything if it's rainy day he's at home and he cooks next dish so he cooks chicken for you so it is like the first input is whether that's sunny or rainy the second input is the dish that he made previously so these input provided to a state will predict the next output it's like a vector product plus the addition of some of the other things that goes in complex neural networks you can just make sense out of it yeah yeah yeah i'll be making it public yeah okay thank you very much again yeah no basically i applied the inverse transform function so yeah yeah yeah yeah it will be inside this range but the results will get inverted so the current value so it will be inverse transform function so that will be adding something divided by something that is lesser so it will be just exploding the value from zero one to scale of say hundreds the yeah yeah yeah it's it's yeah that is why the prediction is not predicting the actual result it is having some variation with the results because you are using some smoothened value and using that smoothened value you are going to predict actual results so these will have variations between thanks
 All right. A very warm welcome and good morning all of you. My name is Anmol  Krishnadevar and the topic for today is understanding and implementing  recurrent neural networks. So before starting let's first see an example. If  we are reading a book and it has say five chapters, we are on chapter one, we  have read that, we are going to chapter two, we are starting reading it, but  suddenly we forgot everything that was there in chapter one. So how will we be  able to understand chapter two? It means short-term memory plays a very crucial  role. On that motion I would like to start with the recurrent neural networks.  So Martin has already introduced me. I'll be skipping this. For  requisites you should be aware of Python language. You should have a decent  knowledge of artificial neural networks and elementary linear algebra. So  recurrent neural networks can be thought of as the neural networks which  persist information and we can think it off as sequential processes, basically  influencing decisions. So in contrast to the traditional neural networks which  don't persist information, recurrent neural networks are having this  advantage over those. So we can think recurrent neural networks as the  networks having loops to themselves. I'll be explaining the architecture now.  And these are one of the most complex supervised deep learning algorithms we  have today. So consider a normal neural network and it has many layers. X naught,  X1, X2 is basically the input that we are providing to the layer. A is a hidden  layer. So the input from X naught goes to A and the input gets transferred from  one hidden layer to another hidden layer. Simultaneously it gets transferred from  the other hidden layer to the next hidden layer and we can think it off  squashing this thing that you see on your right hand side into the thing that  is known as the vanilla RNN. That means it is having loop to itself. RNNs can be  of many architectures. The first one is one to many architecture in which you  provide one input and you can get many outputs. So you can think of, say, you  provide with an image and it provides with certain caption. So an image is mapped  into caption of, say, 506 words. So it is one image getting mapped into 506 words  that is one to many transformation of RNN. Likewise we have many to one. So many  to one can be thought of as, say, we are having video frames and we are generating  text out of the video. So it can be mapped to many to many or many to one. So we  can get from a video a sentence and similarly we can have many to many  transformation also. So recurrent neural networks have the following applications  that is image captioning, subtitle generation, time series classification,  language modeling, natural language processing, even JAPO development is  also based on RNNs nowadays. So there is a major problem in the vanilla RNN. That  is if the RNN is having so many layers, so adjusting the weights at each hidden  layer is a problem. That means once the information from X naught is transferred  to A, it is multiplied by some weight matrix, then transferred to the next layer,  then transferred to next, then transferred to next. And simultaneously the output  gets generated on the top. Now we have something called as loss function. That  is the difference between the actual output and the predicted output. So if the  loss functions for say ht plus one generate some error, so there's some  like error of say 0.15 percent and it's need to be back propagated throughout  the network and back propagating through a network that is so large is  difficult because each time you can think of say W is getting multiplied at each  layer and multiplying W that is between zero and one, multiplying anything by  zero and one, between zero and one say 0.2. If we multiply something by zero  point two multiple times then it will be tending to very small value. So it will  be like the gradient will be propagating through the whole chain but it will be  taking much time to train. So it is not feasible solution. So vanilla RNN  suffers from the vanishing gradient problem. Likewise we have the  exploding gradient problem. So exploding gradient problem is when the value of W  is greater than one. So if you multiply simultaneously a number by say a factor  of more than one then it will tend to always go and multiply to many huge  values. Now the vanishing gradient problem can be thought of as W is less  than one and exploding gradient can be thought of as W is greater than one. To  solve these vanishing gradient problem and the exploding gradient problems we  use certain techniques. So for exploding gradient problem we have truncated  back propagation. So in truncated back propagation we divide the whole set into  certain batches and in those certain batches we just back propagate within  those certain batches sequentially. In system of rewards and penalties it is  like reinforcement learning. So we provide rewards for say if the back  propagation is doing well else we provide penalties. And we have gradient  clipping. If the gradient goes beyond some range then we just clip that  gradient and we don't propagate it through the network. For vanishing gradient  problem we have smart weight initialization that is like a guesswork.  Then we have equestrate networks and we have LSTM. So I'll be talking today of  LSTM that is long short-term memory and it is one of the most used variants of  RNN. So LSTM is one of the most used variants of RNN and the approach for  LSTM is making the weight W equal to one. So you can think of you are not having  W less than one, you are not having W greater than one so what else we can do?  We can just make W equal to one. Now coming to the architecture of the LSTM  C is the new introduction in here. HT is the state of the current input the  hidden layer and CT can be thought of as a cell state. So in here if we provide  the input XT and the hidden layer and we transfer this thing to four states FIG  O. F is the you can say the final gate of forget gate, I is the input gate, G is  another gate, there's no name for it and O is the output gate. We take we apply the  function like the element wise multiplication and the addition using  this formula. So CT is basically F times that is forget gate times the CT minus  one that is the previous state of the cell that was there. It is basically  forgetting some part of the memory. So it is like you were reading something and  you actually forgot some part of it but you retain certain part of it. So this  is actually forgetting major part of it and retaining some part of it and that  some part is used for training the rest of the network and this is actually  applied through the tanh function and the output gate is multiplied and we get  the output of the hidden layer. So comparing the RNN and the LSTM, it is  we can say the LSTM has a sophisticated architecture in terms that it has a  cell state and using this cell state it is like a super highway through which  W is not getting changed. So we are actually changing the element wise  fraction that is F and we are not changing the weight matrix. So multiplying  anything by element or scalar multiplication is much simpler than the  matrix multiplication which is involved in the weight matrix multiplication that  is used in the vanilla RNN. So that is why the vanishing gradient problem goes  off in LSTM. Now we can start building the LSTM and the code I'll be  providing for that. So you can just join in this, you can just implement this as  it is now. I'll be just shifting to the implementation thing. So the major  task of implementing the RNN involves the data preprocessing.  Data preprocessing then building the recurrent neural model and making the  predictions and visualizations. So before implementing we need certain  libraries that is Keras library then we use scikit-learn, we use tensorflow and  yeah. So first task is data preprocessing. So we import the NumPy library, we  import the Matplot library, we import the Pandas library. The NumPy library is for  visualizing the results, is for manipulation Matplot is for visualizing  the results and Pandas is for managing the data sets because the Keras library,  yeah, zoom. Okay.  Can you see now? Okay. So since the Keras library doesn't support the data frames  of Pandas, we need to use the NumPy arrays for Keras library. Now before starting,  let me introduce you to the problem. So today we'll be predicting the stock price  for the Google for first month of 2017. So these are the records that you can see.  These are the records that you can see. So these are 20 records that is each  month has 20 financial days, that means weekdays. So this is the record that we  have to predict on and the training data set is this. So we'll be predicting the  open stock prices of Google for first month of 2017 and we'll be training on  the year 2012 to 2016, so five years of data. Now actually the prediction should  look like this. So if I take these two columns and just insert the plot for it.  So the result that we should predict should be like this. So today our task is to  mimic this behavior of the stock price in January 2017. Now let's get started  So the training data set is I'm calling it Google stock price train.csv. I'm  importing it as a data set, the Pandas data frame. So I'm using pd.readcsv.  Then since this data frame contains many columns but we only require the open  stock prices, the open stock prices that is column one. So I use the iLock function  for selecting, the first parameter in iLock is the column that means it will  be selecting all the rows and the column, the second parameter is the column  parameter that is one to two, one points to the open and two basically points to  the next column but since the second value is omitted so it is just  considering the first column that is the open column and dot value is for  converting it to the NumPy arrays. So we are just converting the Pandas data  frame to the NumPy array. Now we need to scale this data and we'll be using the  scikit-learn preprocessing library and from that we'll be using the minmax  scalar class. So sc is the object of minmax scalar class and the feature range  parameter here tells that you need to convert the values of the column between  the range 0 to 1 so everything will be scaled between 0 to 1 and after that we  just fit and transform. So fit and transform means fitting basically means  normalizing the data so it actually takes the minimum value from that column  and the maximum value from that column and it puts it, the transform puts it  into the normalization function. The normalization function can be the  standardized function or the normal normalization function so in standardized  function we take the mean value and we subtract it from the actual value and  then we divide it by mean value but in normalization function we take the min  value we subtract it from the main value and we divide it by the difference  between the max value and the min value. So it is a basic normalization thing  that we are doing so fit and transform does this thing. After that we need to  create the training data, the training list and the predicted ground truth list.  So X train is a list, an empty list and Y train is an empty list that is for the  training you can say set and the ground truth set. So we have 1 to 5-8  records here in the stock prediction data so we have 1 to 5-8 records these  are 1 to 5-8 records and we are taking the time step as 60 so what is time  step? Basically time step is a very important concept here in the recurrent  neural networks. It is basically how many observations you will be taking into  consideration for training the next value or you can say if you are focusing  on just predicting the ith value then what set of values you will be taking  into consideration before that ith value. So it is basically I'm starting from  the 60th record it means it will be taking the data of three months training  on that and predicting the 61st value. So it is just an assumption you can take  20 days, 18 days, anything but it is like during my testing I took 60 and it  produced great results so I'm using 60 as the time step value and 1 to 5-8 is the  number of records. So I append to the empty list the training scaled values  from the I-60th value to the current value so that is basically appending it  is making an array in which one record contains 60 previous values from the  current value so it is like a 6 you can say n cross 60 matrix that is getting  created. Now the wire train is basically the ground truth and the ground truth  will be just having the current value so it is from I to I plus 1 and 0 is  basically the column that is the first column because we have already cleaned  our data and we are just having one column for open stock prices so 0 points  that thing. Now X train comma Y train equals NP dot array so it is converting  to numpy array so we are generating X train numpy array and Y train numpy array  then we need to reshape the X train array so reshaping means that to work  seamlessly with the you can say numpy arrays and the RN and stuff we need to  add an extra dimension to the this existing numpy array so basically this  is this basically X train dot shape is the number of rows X train dot shape one  is number of columns and one is basically the indicator column that is the  open stock column so we are just reshaping to build it a 3d array from a  2d array. Now the first step of data pre-processing has been done and we are  on to the next step of building the RNN does anyone have any doubt in this yeah  yeah so basically we need to train the data we need to say we are not having  we are having values say 700 800 1000 to say 20 20 to 5 in the stock prices and  it is not ideal it will take a lot of time and it will not generate good  results if the data is not scaled between some value say 0 or 1 so I have  provided the RN 0 to 1 to make it more cleaner so it will be approximated to  the nearest value so it is just cleaning the data and smoothening the data so  scaling has that use yeah  yeah  mm-hmm right so basically the question that you are asking is a very good  question and we use something called grid search for this that is basically a  part of hyper parameter tuning so these are basically pointing towards the  hyper parameter stuff so 60 is like hyper parameter only which is making or  which is amending how the predictions are being made  so what we can do is after this is done like this is the first school  for you all people but I tried it with certain parameters and in the hyper  parameter tuning the grid search library provides you with you can say you can  provide arrays of values say I have something called say time step so I  provide time step equal to 30 comma 60 comma 300 suppose  and I have another value say optimizer and I take three optimizers  suppose which are used for this type of regression this is a regression problem  because we are dealing with the continuous values so we take say rms  prop optimizer that I'll be telling you more about and we take atom optimizer  and we say n atom optimizer so it will be doing a cross product so it will be  take this parameter 30 time step cross it with the  the first rms prop and generate result then 30 add them generate result 30 and  atom generate result then it will be doing same thing with 60 rms prop 60  add them 60 and add them so it will be generating  nine results and from those nine results you can  actually get the matrix formed which will be showing  which one is generating good accuracy and less loss  so so using that grid search thing you can actually judge which hyper parameter  to tune to what value so that is part of grid search  thanks  no I have not because it is like I don't have that much time for showing  yeah so now building the RNN stuff so we are importing four classes  sequential dense lstm and dropout sequential class is for basically  inputting the first you can say set of inputs to the neural network  dense class is for the last layer of the neural network that's the output layer  lstm class is for making the hidden layers of the neural network  and dropout is for something called as dropout regularization which I'll be  telling you more about further so we are making a regressor since it's not  a classification problem it's we are dealing with the continuous  data set so it's a regression problem so we make an object of sequential class  called regressor and we add the first layer to it so it's regressor.add so  regressor.add will add layers to the neural network so first layer is the  lstm layer the hidden layer that we are adding and  units equal to 50 means the number of neurons you can say  that we are introducing to the network so this layer will have 50 neurons  and you can take it any value so it was again the hyper parameter stuff so I  took 50 then return sequences equal to true means  that the output of this layer will be getting forwarded to the next layer  and input shape equals x-ray in dot shape one that points to the  columns of the data set and one is for reshaping it to the third dimension  so we are just using the the column and the third dimensional value for it  we don't need to use the first row dimension value for it  so this will be adding the first layer and adding the first layer  may introduce overfitting say so to deal with the overfitting problem  that is to say avoid the noise that is being created by the overfitting  stuff we use the dropout regularization stuff so dropout  actually means dropping out certain number of neurons from the layer  so out of 50 I have chosen 0.2 that's 20 percent to be dropped out so it means  it will be just considering 40 neurons for the next layer  and out of those 50 neurons 10 neurons at random or say 20 percent of those  50 neurons will be dropped out at random so this actually  avoids the overfitting problem yeah so basically it's a stacked del S T M so  I am using four hidden layers in this and for the four layers I'm adding four  dropouts so yeah  mm-hmm  yeah  no no it's like during the epochs epochs is basically how many time the data the  full data will be propagated forward and backward to  lessen the loss the loss generated by you can say the gradient  so it's actually for the number of epochs the dropout will be done at random  so dropping out means just not considering 20 percent of those neurons  randomly in one epoch and in the next epoch it will again be choosing random  20 percent and then be not considering so it's like a random stuff going on I  chosen 100 epochs so it's fully propagating the  data forward and backward and it training is done 100 times on this data  so it's like for each epoch it is generating some  random value some it's choosing some random neuron and it's dropping out  10 random neurons so it actually avoids the overfitting problem in that  way yeah and then similarly I'm adding three more  layers so this is regressor.add LSTM now you have added the first  layer initially so you don't need to provide the input shape because LSTM  class already knows that what is the input so from the second layer you don't  need to provide this parameter input shape it will automatically take  from the previous thing and this is the second layer then this is the third  layer and the fourth layer in fourth layer we don't actually need the output  return sequence because we need to pass it through the dense layer and not  another LSTM layer so we'll be omitting the return sequence  equal to true parameter we can state here return sequence equal to false but  the default value of return sequence is false so we are not putting here the  parameter and dense is the final output  layer so basically it is the unit is equal to one means it will be having  just one neuron that's the resultant neuron so dense  is actually corresponding to the result of the neural network  now the building of RNN has been done but the compilation phase is still lift  compilation actually means compiling with a certain  loss function and choosing the right optimizer so as i told you earlier  rms prop and adam are the two optimizers that are used  with RNN and adam is the one that usually works  apart from say rn also it works with cnn also  so it is a much wider optimizer that you can use  and it gives good results again a hyper parameter  tuning thing so i have chosen the optimizer as adam  and since it is the regression stuff so we'll be using mean squared error  loss function if it would have been  say the classification stuff then we would have used the cross entropy  or binary cross entropy loss function then  we have compiled this and then we are ready to fit it  so fitting the RNN for the training set so we pass the fit function x-train  y-train epochs equal to 100 batch sizes 32 so it will be taking batch of 32 it  will be dividing the data set into 32  say element batches and it will be working on those  batches it's again hyper parameter tuning stuff so  what i mean by hyper parameter tuning is that tuning the parameters  of the classes in such a way that it provides  good result or good predictions now the task tree is the prediction task so  we have done the RNN building so we have done the data preprocessing  stuff now is the task to predict the results so actually  the data set for predicting was separate than the data set for testing  so we just take the test data set and we merge  that test data set with the actual prediction data set to make it  a total data set which will contain every value  so here we are just converting the read csv we are creating a data frame of the  test data set and then we are again converting it to the  numpy array as we did it with the price the prediction price  stuff then we concatenate the open column of the prediction data set with the  test column so it will be 1, 2, 5, 8 plus 20 records that's  1, 2, 7, 8 records in total because during the time of testing we need to  have we need to consider the previous 60 records and  for considering those previous 60 records we should have a data set  incomplete so i'm just using this complete data set  and x is zero is for vertical join input equals data set merged so  basically it is taking it is creating another  you can say numpy array which is based on the merge data set and it will be  taking the values 60 values from the current state so  counting from 1, 2, 5, 8 record number 1, 2, 5, 8 it will be taking into account  the previous 60 values to predict the results of  the record 1, 2, 5, 8 now we are reshaping it again and since we have  already fitted the data earlier we'll just be transforming it  so we are just taking the input numpy array and we are transforming the input  numpy array for making the predictions and we use another list called x test  and we append to this list the values from the last 20 rows that we need to test  and then we form the numpy array of this and we just shape it again reshape it  again now the prediction can be done using the  predict function of the regressor class the  sequence the class that we formed so regressor.predict  basically makes prediction on the x test list that we just formed  the numpy array and inverse transform is the function  that is used to inverse the normalization stuff that we did so  now we are just you can say we are inversing the scaling that we did  we made the scaling from 0 to 1 we normalized the results from 0 to 1  we are just inversing the transform to actually convert them into the  real value so it will be something like 0 point something is converted into 770  that is say a stock price for one record and then we plot these results so  since the time is short i have already run this thing  and paacham so it was say 100 epochs and you can see that i'll just  show you say  you see that with increasing number of epochs the loss function is decreasing  so the value of loss is decreasing so the loss must have started with say  some value 0.3 or something 0.0034 and it's now at the end of 100 epochs it's  0.001315 so you can see as i'm going up the loss  function is increasing so the value of loss function is increasing  it means that the training has been done correctly and we have achieved certain  level of accuracy so that the loss function has  minimized its value from say it's in 30s or 40s  say 0.0030 to 0.13 and we can see that  the plot that we have  is following the trend of ups and downs of the market but it's not actually  predicting the correct values though it is following the trend  that's like if the value of the stock price is going up it is  following that trend so blue is the predicted one and red is the actual one  so using these parameters i was able to generate this result doing  hyper parameter tuning with grid search will definitely  improvise these results and it will actually match with  some error with the actual red line so it is like just following the trend  with the red thing so we can see that it is actually following  following the trend so we can see on x-axis at  point 2.5 it is just going down then it's following the upward trend and then  it's being stable at the end so it was this prediction that  using these hyper parameters i was able to make  so the source code for this i'll be updating my slides  but it's on  it's on github  it's at this address i'll be just uploading my slides and it will get  uploaded on the session page and  after that so these are my acknowledgements  professor martin christian then my supervisors christopher wula  for creating good blogs and yeah if you have any questions then  you can ask yeah thanks  thank you very much we have time for a couple of questions  thank you i wanted to ask how did you tune your architecture why four layers  okay so when we deal with RNNs we cannot go beyond say we generally choose two  three or four layers because it actually doesn't  make any sense going beyond four layers as the results  will not improve over choosing four layers so  for just showing the implementation stuff i used four layers  and it was just making some good assumptions  when compared with using three layers or two layers  but generally in RNNs we just don't go beyond four  layers it can be the case that if we are using RNNs with  some other neural network say CNN for say image captioning task  then it will be another scenario we'll be getting the results of the vector  generated from the CNN and we'll be transferring those CNN  results into the first hidden layer that's the  H0 thing and those results will be just making further  you can say vectors forming vectors which will then be propagated throughout  the chain so it will be having more impact on the RNN training and  in that case we can just lower down from four layers to two layers because it is  already having lots of information from the CNN stuff  so it is basically based on that thing so if we consider a basic example  that's the perfect roommate example that is very famous in field of RNN  if you have a roommate who cooks food for you and he cooks  say apple pie chicken and say anything xyz anything  so on sunny day he cooks say apple pie and if it's another sunny day then he  just goes off the duty and doesn't cook anything if it's  rainy day he's at home and he cooks next dish so he  cooks chicken for you so it is like the first input is whether that's sunny or  rainy the second input is the dish that he made previously  so these input provided to a state will predict the next  output it's like a vector product plus the addition of some of the other  things that goes in complex neural networks you can  just make sense out of it yeah  yeah yeah i'll be making it public yeah okay thank you very much again  yeah  no basically i applied the inverse transform function  so  yeah yeah yeah yeah it will be inside this range but the results will get  inverted so the current value so it will be inverse transform function  so that will be adding something divided by something that is lesser  so it will be just exploding the value from zero one to scale of say hundreds  the yeah yeah yeah it's it's yeah that is why the  prediction is not predicting the actual result it is having  some variation with the results because you are using some smoothened value and  using that smoothened value you are going to predict  actual results so these will have variations between  thanks
Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_0_143.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_144_1043.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_1044_14099.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_14100_14579.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_14580_14639.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_14640_14699.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_14700_15167.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_15168_15971.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_15972_16019.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16020_16091.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16092_16223.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16224_16619.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16620_16667.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16668_16751.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16752_16859.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16860_16967.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_16968_17039.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_17040_17123.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_17124_17483.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_17484_17567.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_17568_18215.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_18216_18347.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_18348_18743.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_18744_18803.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_18804_19343.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_19344_19391.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_19392_19475.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_19476_20435.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_20436_23219.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_23220_23471.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_23472_39827.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_39828_43811.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_43812_43859.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_43860_45659.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_45660_45731.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_45732_48047.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_48048_49859.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_49860_49943.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_49944_51623.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_51624_51683.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_51684_51731.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_51732_51779.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_51780_51887.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_51888_51947.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_51948_52007.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52008_52067.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52068_52115.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52116_52415.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52416_52463.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52464_52619.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52620_52679.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52680_52895.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_52896_53171.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_53172_53351.png Anmol_Krishan_Sachdeva_-_Understanding_and_Implementing_Recurrent_Neural_Net_53352_59458.png