<jupyter_start><jupyter_text>Loading Data<jupyter_code>def get_stock_data(stock_name,start_date,end_date,normalized=0):
    data = yf.download(stock_name,start_date,end_date)
    df = pd.DataFrame(data)
    df.drop(['Adj Close'],inplace=True,axis=1)
    return df
stock_name = 'GOOGL'
df = get_stock_data(stock_name,'2013-07-12','2019-07-11',0)
df.tail()
from matplotlib import pyplot as plt
plt.figure(figsize=(20,10))
plt.plot(df["Open"])
plt.plot(df["High"])
plt.plot(df["Close"])
plt.title('Google stock price history')
plt.ylabel('Price (USD)')
plt.xlabel('Days')
plt.legend(['Open','High','Close'], loc='upper left')
plt.show()
plt.figure(figsize=(20,6))
plt.plot(df["Volume"])
plt.title('Google stock volume history')
plt.ylabel('Volume')
plt.xlabel('Days')
plt.show()
# We can observe that during mid of 2013, their is a clear surge in Volume, maybe due to low price.
sns.scatterplot(df['Volume'],df["Open"])
# We can irrespective of price google is a well traded stock
print("checking if any null values are present\n") 
df.isnull().sum()<jupyter_output>checking if any null values are present<jupyter_text>Normalizing the data    The data is not normalized and the range for each column varies, especially Volume. Normalizing data helps the algorithm in converging i.e. to find local/ global minimum efficiently.<jupyter_code>train_cols = ["Open","High","Low","Close","Volume"]
df_train, df_test = train_test_split(df, train_size=0.8, test_size=0.2, shuffle=False)
print("Train and Test size", len(df_train), len(df_test))
# scale the feature MinMax, build array
x = df_train.loc[:,train_cols].values
min_max_scaler = MinMaxScaler()
x_train = min_max_scaler.fit_transform(x)
x_test = min_max_scaler.transform(df_test.loc[:,train_cols])<jupyter_output>Train and Test size 1207 302<jupyter_text>Supervised Learning and time series data    LSTMs consume input in format [ batch_size, time_steps, Features ]; a 3- dimensional array.    A) Batch Size:    Used to say how many samples of input do you want the Neural Net to see before updating the weights. So let’s say you have 100 samples (input dataset) and you want to update weights every time your NN has seen an input. In that case batch size would be 1 and total number of batches would be 100.    As it turns out using very small batch size reduces the speed of training and on the other hand using too big batch size (like whole dataset) reduces the models ability to generalize to different data and it also consumes more memory.     B) Time Steps:    Define how many units back in time you want your network to see. For example if you were working on a character prediction problem where you have a text corpus to train on and you decide to feed your network 6 characters at a time. Then your time step is 6. In our case we will be using 60 as time step i.e. we will look into 2 months of data to predict next days price.    C) Features:    Is the number of attributes used to represent each time step. Consider the character prediction example above, and assume that you use a one-hot encoded vector of size 100 to represent each character. Then feature size here is 100.In our case lets use the time steps to be 4, using 4 days our network can be used to predict the 5th day.Lets use the batch size to be 2, in order to update the weights accordingly.Features we are using as input are ["Open" ,"High" ,"Low" ,"Close" ,"Volume"], so output will also be the same features using a particular batch size to update the weights and we use 4 day's are input to predict the 5th day's close price.  Data Preparation<jupyter_code>from tqdm import tqdm_notebook
def build_timeseries(mat, y_col_index,TIME_STEPS):
    # y_col_index is the index of column that would act as output column
    # total number of time-series samples would be len(mat) - TIME_STEPS
    dim_0 = mat.shape[0] - TIME_STEPS
    dim_1 = mat.shape[1]
    x = np.zeros((dim_0, TIME_STEPS, dim_1))
    y = np.zeros((dim_0,))
    
    for i in tqdm_notebook(range(dim_0)):
        x[i] = mat[i:TIME_STEPS+i]
        y[i] = mat[TIME_STEPS+i, y_col_index]
    print("length of time-series i/o",x.shape,y.shape)
    return x, y
def trim_dataset(mat, batch_size):
    """
    trims dataset to a size that's divisible by BATCH_SIZE
    """
    no_of_rows_drop = mat.shape[0]%batch_size
    if(no_of_rows_drop > 0):
        return mat[:-no_of_rows_drop]
    else:
        return mat
# x,y <- build_timeseries(matrix_input,outcolindex(close price),Time_steps(as input))
x_t, y_t = build_timeseries(x_train, 3,2)
BATCH_SIZE = 2
x_t = trim_dataset(x_t, BATCH_SIZE)
y_t = trim_dataset(y_t, BATCH_SIZE)
x_temp, y_temp = build_timeseries(x_test, 3,2)
x_val, x_test_t = np.split(trim_dataset(x_temp, BATCH_SIZE),2)
y_val, y_test_t = np.split(trim_dataset(y_temp, BATCH_SIZE),2)<jupyter_output><empty_output><jupyter_text>Building Model<jupyter_code>BATCH_SIZE = 2
TIME_STEPS = 2
lstm_model = Sequential()
lstm_model.add(LSTM(100, batch_input_shape=(BATCH_SIZE, TIME_STEPS, x_t.shape[2]), dropout=0.0, recurrent_dropout=0.0, stateful=True,     kernel_initializer='random_uniform'))
lstm_model.add(Dropout(0.5))
lstm_model.add(Dense(20,activation='relu'))
lstm_model.add(Dense(1,activation='sigmoid'))
optimizer = tf.keras.optimizers.RMSprop(lr=0.0001)
lstm_model.compile(loss='mean_squared_error', optimizer=optimizer)<jupyter_output><empty_output><jupyter_text>Hyper-parameter tunning<jupyter_code>es = EarlyStopping(monitor='val_loss', mode='min', verbose=1,
                       patience=40, min_delta=0.0001)
              
from matplotlib import pyplot as plt

# Keras fit returns keras.callbacks.history object which stores loss info
history = lstm_model.fit(x_t,y_t, epochs=params["epochs"], verbose=2, batch_size=BATCH_SIZE, 
              shuffle=False, validation_data=(trim_dataset(x_val, BATCH_SIZE), 
              trim_dataset(y_val, BATCH_SIZE)), callbacks=[es])

plt.figure()
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()
# model.evaluate(x_test_t, y_test_t, batch_size=BATCH_SIZE
y_pred = lstm_model.predict(trim_dataset(x_test_t, BATCH_SIZE), batch_size=BATCH_SIZE)
y_pred = y_pred.flatten()
y_test_t = trim_dataset(y_test_t, BATCH_SIZE)
error = mean_squared_error(y_test_t, y_pred)
print("Error is", error, y_pred.shape, y_test_t.shape)
print(y_pred[0:15])
print(y_test_t[0:15])
# convert the predicted value to range of real data
y_pred_org = (y_pred * min_max_scaler.data_range_[3]) + min_max_scaler.data_min_[3]
# min_max_scaler.inverse_transform(y_pred)
y_test_t_org = (y_test_t * min_max_scaler.data_range_[3]) + min_max_scaler.data_min_[3]
# min_max_scaler.inverse_transform(y_test_t)
print(y_pred_org[0:15])
print(y_test_t_org[0:15])

# Visualize the training data
from matplotlib import pyplot as plt
plt.figure()
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()
error = mean_squared_error(y_test_t, y_pred)
print("Error is", error, y_pred.shape, y_test_t.shape)
# Visualize the prediction
from matplotlib import pyplot as plt
plt.figure()
plt.plot(y_pred_org)
plt.plot(y_test_t_org)
plt.title('Prediction vs Real Stock Price')
plt.ylabel('Price')
plt.xlabel('Days')
plt.legend(['Prediction', 'Real'], loc='upper left')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1129226246157958, 'fraction_numerical': 0.01710543899505546, 'mean_word_length': 5.490893321769297, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '476734', 'license': 'MIT', 'n_tokens_mistral': 2709, 'n_tokens_neox': 2545, 'n_words': 740, 'path': 'LSTM for Stock prediction/stock_prediction_Lstm.ipynb', 'repo_name': 'swaroop9ai9/Fintech', 'size': '7483'}