<jupyter_start><jupyter_text>Need to test if add features are benefitials<jupyter_code>df = pd.read_csv('../data/num_data.csv')
dataset = df.sample(1000)
dataset.shape
dataset.head()
# Some functions to help out with
def plot_predictions(test, predicted):
    plt.figure(figsize=(30, 15));

    plt.plot(test, color='red', label='Real PM2.5 Concentration',)
    plt.plot(predicted, color='blue', alpha=0.5, label='Predicted PM2.5 Concentation')
    plt.title('PM2.5 Concentration Prediction')
    plt.xlabel('Time')
    plt.ylabel('PM2.5  Concentration')
    plt.legend()
    plt.show()
    

def return_rmse(test,predicted):
    rmse = math.sqrt(mean_squared_error(test, predicted))
    print("The root mean squared error is {}.".format(rmse))
train_size=int(dataset.shape[0] * 0.6)
test_size = int(dataset.shape[0] - train_size)
training_set = dataset[:train_size].iloc[:,4:5].values
test_set = dataset[train_size:].iloc[:,4:5].values
training_set.shape
# # Scaling the training set
sc = MinMaxScaler(feature_range=(0,1))
training_set_scaled = sc.fit_transform(training_set)
# Since LSTMs store long term memory state, we create a data structure with 24 timesteps and 1 output
# So for each element of training set, we have 60 previous training set elements 
X_train = []
y_train = []
seq_len = 60
for i in range(seq_len, train_size):
    X_train.append(training_set_scaled[i-seq_len:i,0])
    y_train.append(training_set_scaled[i,0])
X_train, y_train = np.array(X_train), np.array(y_train)
X_train.shape
# Reshaping X_train for efficient modelling
X_train = np.reshape(X_train, (X_train.shape[0],X_train.shape[1],1))
X_train.shape
(X_train.shape[1],1)
GRU_reg = Sequential()
LSTM_reg = Sequential()
GRU_GRU_reg =  Sequential()
GRU_LSTM_reg = Sequential()
LSTM_GRU_reg = Sequential()
LSTM_LSTM_reg = Sequential()

GRU_reg.add(GRU(units=0))
GRU_reg.add(Dropout(0.2))
# The output layer
GRU_reg.add(Dense(units=1))


LSTM_reg.add(LSTM(units=50, return_sequences=True))
# The output layer
LSTM_reg.add(Dense(units=1))


LSTM_GRU_reg.add(LSTM(units=100, return_sequences=True, input_shape=(X_train.shape[1],1)))
LSTM_GRU_reg.add(GRU(units=100, activation='tanh'))
# The output layer
LSTM_GRU_reg.add(Dense(units=1))


GRU_LSTM_reg.add(GRU(units=50, return_sequences=True, input_shape=(X_train.shape[1],1), activation='tanh'))
GRU_LSTM_reg.add(LSTM(units=50, activation='tanh'))
# The output layer
GRU_LSTM_reg.add(Dense(units=1))

# Compiling the RNNs
sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)

GRU_reg.compile(optimizer='rmsprop',loss='mean_squared_error')
LSTM_reg.compile(optimizer='rmsprop',loss='mean_squared_error')

LSTM_GRU_reg.compile(optimizer=sgd,loss='mean_squared_error')

GRU_LSTM_reg.compile(optimizer='rmsprop',loss='mean_squared_error')
# Fitting to the training set
# LSTM_reg.fit(X_train,y_train,epochs=50,batch_size=32)
# GRU_reg.fit(X_train,y_train,epochs=50,batch_size=32)
# LSTM_GRU_reg.fit(X_train,y_train,epochs=50,batch_size=32)

# LSTM_GRU_reg.fit(X_train,y_train,epochs=50,batch_size=32)
LSTM_GRU_reg.fit(X_train,y_train,epochs=50,batch_size=32)
# Now to get the test set ready in a similar way as the training set.
# The following has been done so forst 60 entires of test set have 60 previous values which is impossible to get unless we take the whole 
# 'High' attribute data for processing
inputs = dataset[len(dataset)-len(test_set) - seq_len:].values
inputs = inputs.reshape(-1,1)
inputs  = sc.transform(inputs)

# Preparing X_test and predicting the prices
X_test = []
for i in range(seq_len,test_size+seq_len):
    X_test.append(inputs[i-seq_len:i,0])
X_test = np.array(X_test)
X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1))
predicted_PM25 = LSTM_GRU_reg.predict(X_test)
predicted_PM25 = sc.inverse_transform(predicted_PM25)

y_train_pred = LSTM_GRU_reg.predict(X_train)
y_train_pred = sc.inverse_transform(y_train_pred)
plot_predictions(training_set,y_train_pred)
# Visualizing the results for LSTM
plot_predictions(test_set,predicted_PM25)
# Evaluating our model
return_rmse(test_set,predicted_PM25)<jupyter_output>The root mean squared error is 88.22507466163962.<jupyter_text>Truth be told. That's one awesome score. LSTM is not the only kind of unit that has taken the world of Deep Learning by a storm. We have **Gated Recurrent Units(GRU)**. It's not known, which is better: GRU or LSTM becuase they have comparable performances. GRUs are easier to train than LSTMs. Gated Recurrent UnitsIn simple words, the GRU unit does not have to use a memory unit to control the flow of information like the LSTM unit. It can directly makes use of the all hidden states without any control. GRUs have fewer parameters and thus may train a bit faster or need less data to generalize. But, with large data, the LSTMs with higher expressiveness may lead to better results.They are almost similar to LSTMs except that they have two gates: reset gate and update gate. Reset gate determines how to combine new input to previous memory and update gate determines how much of the previous state to keep. Update gate in GRU is what input gate and forget gate were in LSTM. We don't have the second non linearity in GRU before calculating the outpu, .neither they have the output gate.Source: [Quora](https://www.quora.com/Whats-the-difference-between-LSTM-and-GRU-Why-are-GRU-efficient-to-train)<jupyter_code># The GRU architecture
regressorGRU = Sequential()
# First GRU layer with Dropout regularisation
regressorGRU.add(GRU(units=50, return_sequences=True, input_shape=(X_train.shape[1],1), activation='tanh'))
regressorGRU.add(Dropout(0.2))
# Second GRU layer
regressorGRU.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1],1), activation='tanh'))
regressorGRU.add(Dropout(0.2))
# Third GRU layer
regressorGRU.add(GRU(units=50, return_sequences=True, input_shape=(X_train.shape[1],1), activation='tanh'))
regressorGRU.add(Dropout(0.2))
# Fourth GRU layer
regressorGRU.add(GRU(units=50, activation='tanh'))
regressorGRU.add(Dropout(0.2))
# The output layer
regressorGRU.add(Dense(units=1))
# Compiling the RNN
regressorGRU.compile(optimizer=SGD(lr=0.01, decay=1e-7, momentum=0.9, nesterov=False),loss='mean_squared_error')
# Fitting to the training set
regressorGRU.fit(X_train,y_train,epochs=50,batch_size=150)<jupyter_output><empty_output><jupyter_text>The current version version uses a dense GRU network with 100 units as opposed to the GRU network with 50 units in previous version<jupyter_code># Preparing X_test and predicting the prices
X_test = []
for i in range(60,311):
    X_test.append(inputs[i-60:i,0])
X_test = np.array(X_test)
X_test = np.reshape(X_test, (X_test.shape[0],X_test.shape[1],1))
GRU_predicted_stock_price = regressorGRU.predict(X_test)
GRU_predicted_stock_price = sc.inverse_transform(GRU_predicted_stock_price)
# Visualizing the results for GRU
plot_predictions(test_set,GRU_predicted_stock_price)
# Evaluating GRU
return_rmse(test_set,GRU_predicted_stock_price)<jupyter_output><empty_output><jupyter_text>Sequence GenerationHere, I will generate a sequence using just initial 60 values instead of using last 60 values for every new prediction. **Due to doubts in various comments about predictions making use of test set values, I have decided to include sequence generation.** The above models make use of test set so it is using last 60 true values for predicting the new value(I will call it a benchmark). This is why the error is so low. Strong models can bring similar results like above models for sequences too but they require more than just data which has previous values. In case of stocks, we need to know the sentiments of the market, the movement of other stocks and a lot more. So, don't expect a remotely accurate plot. The error will be great and the best I can do is generate the trend similar to the test set.I will use GRU model for predictions. You can try this using LSTMs also. I have modified GRU model above to get the best sequence possible. I have run the model four times and two times I got error of around 8 to 9. The worst case had an error of around 11. Let's see what this iterations.The GRU model in the previous versions is fine too. Just a little tweaking was required to get good sequences. **The main goal of this kernel is to show how to build RNN models. How you predict data and what kind of data you predict is up to you. I can't give you some 100 lines of code where you put the destination of training and test set and get world-class results. That's something you have to do yourself.**<jupyter_code># Preparing sequence data
initial_sequence = X_train[2708,:]
sequence = []
for i in range(251):
    new_prediction = regressorGRU.predict(initial_sequence.reshape(initial_sequence.shape[1],initial_sequence.shape[0],1))
    initial_sequence = initial_sequence[1:]
    initial_sequence = np.append(initial_sequence,new_prediction,axis=0)
    sequence.append(new_prediction)
sequence = sc.inverse_transform(np.array(sequence).reshape(251,1))
# Visualizing the sequence
plot_predictions(test_set,sequence)
# Evaluating the sequence
return_rmse(test_set,sequence)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09885308574549427, 'fraction_numerical': 0.022719825232113598, 'mean_word_length': 6.226519337016574, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '129828', 'license': 'MIT', 'n_tokens_mistral': 3219, 'n_tokens_neox': 3053, 'n_words': 903, 'path': 'notebooks/keras_rnn.ipynb', 'repo_name': 'harryli18/hybrid-rnn-models', 'size': '9155'}