<jupyter_start><jupyter_text>Splitting Data into a Training set and a Test set<jupyter_code># First calculate the mid prices from the highest and lowest

high_prices = df.loc[:,'High'].values
low_prices = df.loc[:,'Low'].values
mid_prices = (high_prices+low_prices)/2.0
train_data = mid_prices[:5000]
test_data = mid_prices[5000:]<jupyter_output><empty_output><jupyter_text>Normalizing the Data<jupyter_code># Scale the data to be between 0 and 1
# When scaling remember! You normalize both test and train data with respect to training data
# Because you are not supposed to have access to test data
scaler = MinMaxScaler()
train_data = train_data.reshape(-1,1)
test_data = test_data.reshape(-1,1)
# Train the Scaler with training data and smooth data
scaler = MinMaxScaler()
smoothing_window_size = 1100  # Changed 
for di in range(0, len(train_data), smoothing_window_size):   
     scaler.fit(train_data[di:di + smoothing_window_size, :])
     train_data[di:di + smoothing_window_size, :] = scaler.transform(
      train_data[di:di + smoothing_window_size, :])
   
# Reshape both train and test data
train_data = train_data.reshape(-1)

# Normalize test data
test_data = scaler.transform(test_data).reshape(-1)<jupyter_output><empty_output><jupyter_text>Note that you should only smooth training data.<jupyter_code># Now perform exponential moving average smoothing
# So the data will have a smoother curve than the original ragged data
EMA = 0.0
gamma = 0.1
for ti in range(5000):
    EMA = gamma*train_data[ti] + (1-gamma)*EMA
    train_data[ti] = EMA

# Used for visualization and test purposes
all_mid_data = np.concatenate([train_data,test_data],axis=0)<jupyter_output><empty_output><jupyter_text>So what do the above graphs (and the MSE) say?It seems that it is not too bad of a model for very short predictions (one day ahead). Given that stock prices don't change from 0 to 100 overnight, this behavior is sensible. Next, you will look at a fancier averaging technique known as exponential moving average. Exponential Moving Average formulaIn the exponential moving average method, you calculate xt+1 as,x_t+1=EMA_t=γ×EMAt−1+(1−γ)xt where EMA0=0 and EMA is the exponential moving average value you maintain over time.The above equation basically calculates the exponential moving average from t+1 time step and uses that as the one step ahead prediction. γ decides what the contribution of the most recent prediction is to the EMA. For example, a γ=0.1 gets only 10% of the current value into the EMA. Because you take only a very small fraction of the most recent, it allows to preserve much older values you saw very early in the average. See how good this looks when used to predict one-step ahead below.<jupyter_code>#Calculate the MSE for the EMA
window_size = 100
N = train_data.size

run_avg_predictions = []
run_avg_x = []

mse_errors = []

running_mean = 0.0
run_avg_predictions.append(running_mean)

decay = 0.5

for pred_idx in range(1,N):

    running_mean = running_mean*decay + (1.0-decay)*train_data[pred_idx-1]
    run_avg_predictions.append(running_mean)
    mse_errors.append((run_avg_predictions[-1]-train_data[pred_idx])**2)
    run_avg_x.append(date)

print('MSE error for EMA averaging: %.5f'%(0.5*np.mean(mse_errors)))
#display the MSE of the EMA
plt.figure(figsize = (18,9))
plt.plot(range(df.shape[0]),all_mid_data,color='b',label='True')
plt.plot(range(0,N),run_avg_predictions,color='orange', label='Prediction')
#plt.xticks(range(0,df.shape[0],50),df['Date'].loc[::50],rotation=45)
plt.xlabel('Date')
plt.ylabel('Mid Price')
plt.legend(fontsize=18)
plt.show()
#importing required libraries
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM
#setting index as date
df['Date'] = pd.to_datetime(df.Date,format='%Y-%m-%d')
df.index = df['Date']
print(df.head(6))
#plot
plt.figure(figsize=(16,8))
plt.plot(df['Close'], Label='Close Price history')
data = df.sort_index(ascending=True, axis=0)

new_data = pd.DataFrame(index=range(0,len(df)),columns=['Date', 'Close'])
for i in range(0,len(data)):
    new_data['Date'][i] = data['Date'][i]
    new_data['Close'][i] = data['Close'][i]
#setting index
new_data.index = new_data.Date
new_data.drop('Date', axis=1, inplace=True)
#creating train and test sets
dataset = new_data.values
train = dataset[0:1000,:]
valid = dataset[1000:,:]
#converting dataset into x_train and y_train
scaler = MinMaxScaler(feature_range=(0, 1))
scaled_data = scaler.fit_transform(dataset)

x_train, y_train = [], []
for i in range(60,len(train)):
    x_train.append(scaled_data[i-60:i,0])
    y_train.append(scaled_data[i,0])
x_train, y_train = np.array(x_train), np.array(y_train)

x_train = np.reshape(x_train, (x_train.shape[0],x_train.shape[1],1))
print(x_train.shape)<jupyter_output>(940, 60, 1)<jupyter_text>Define the LSTM Recurrent Neural Network. Here, you can add more LSTM layers and adjust the dropout in order to improve the accuracy of the model.<jupyter_code># create and fit the LSTM network
regressor = Sequential()
regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (x_train.shape[1], 1)))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.2))
regressor.add(Dense(units = 1))

print(regressor.summary())<jupyter_output>Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 lstm (LSTM)                 (None, 60, 50)            10400     
                                                                 
 dropout (Dropout)           (None, 60, 50)            0         
                                                                 
 lstm_1 (LSTM)               (None, 60, 50)            20200     
                                                                 
 dropout_1 (Dropout)         (None, 60, 50)            0         
                                                                 
 lstm_2 (LSTM)               (None, 60, 50)            20200     
                                                                 
 dropout_2 (Dropout)         (None, 60, 50)            0         
                                                        [...]<jupyter_text>Compile and train the model defined in the above step. Iteratively, you can increase or decrease the epochs and batch size to get more accuracy.<jupyter_code>regressor.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics=['accuracy'])
regressor.fit(x_train, y_train, epochs = 15, batch_size = 32)<jupyter_output>Epoch 1/15
30/30 [==============================] - 61s 286ms/step - loss: 0.0025 - accuracy: 0.0000e+00
Epoch 2/15
30/30 [==============================] - 9s 302ms/step - loss: 4.1564e-04 - accuracy: 0.0000e+00
Epoch 3/15
30/30 [==============================] - 9s 303ms/step - loss: 2.9757e-04 - accuracy: 0.0000e+00
Epoch 4/15
30/30 [==============================] - 10s 322ms/step - loss: 2.6436e-04 - accuracy: 0.0000e+00
Epoch 5/15
30/30 [==============================] - 9s 315ms/step - loss: 2.5629e-04 - accuracy: 0.0000e+00
Epoch 6/15
30/30 [==============================] - 9s 304ms/step - loss: 2.2057e-04 - accuracy: 0.0000e+00
Epoch 7/15
30/30 [==============================] - 9s 310ms/step - loss: 2.3546e-04 - accuracy: 0.0000e+00
Epoch 8/15
30/30 [==============================] - 10s 319ms/step - loss: 2.5756e-04 - accuracy: 0.0000e+00
Epoch 9/15
30/30 [==============================] - 9s 306ms/step - loss: 2.2971e-04 - accuracy: 0.0000e+00
Epoch 10/15
30/30 [==========[...]<jupyter_text>Now, our model is trained and needs to be tested on the testing data. For this purpose, fetch the new data for a different period. Preprocessing steps are similar as we have done with training data.<jupyter_code>import datetime
from pandas_datareader import data as pdr
import yfinance as yf
# Importing necessary paackages
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

import re
import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.stem.wordnet import WordNetLemmatizer
import string

from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score, GridSearchCV
from sklearn.metrics import confusion_matrix, classification_report, accuracy_score, f1_score
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.pipeline import Pipeline

seed = 1000
ticker = yf.Ticker("MSFT")

testdataframe= ticker.history(start="2017-01-01", end="2022-01-11")
testdataframe['Date'] = testdataframe.index
testdata = pd.DataFrame(columns = ['Date', 'Open', 'High', 'Low', 'Close'])
testdata['Date'] = testdataframe['Date']
testdata['Open'] = testdataframe['Open']
testdata['High'] = testdataframe['High']
testdata['Low'] = testdataframe['Low']
testdata['Close'] = testdataframe['Close']
real_stock_price = testdata.iloc[:, 1:2].values
dataset_total = pd.concat((df['Open'], testdata['Open']), axis = 0)
inputs = dataset_total[len(dataset_total) - len(testdata) - 60:].values
inputs = inputs.reshape(-1,1)
inputs = scaler.transform(inputs)
X_test = []
for i in range(60, 1300):
    X_test.append(inputs[i-60:i, 0])
X_test = np.array(X_test)
X_test = np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))<jupyter_output><empty_output><jupyter_text>Test the LSTM model on the new dataset<jupyter_code>predicted_stock_price = regressor.predict(X_test)
predicted_stock_price = scaler.inverse_transform(predicted_stock_price)<jupyter_output><empty_output><jupyter_text>Visualize the predicted stock prices with original stock prices<jupyter_code>plt.figure(figsize=(20,10))
plt.plot(real_stock_price, color = 'green', label = 'MSFT Stock Price')
plt.plot(predicted_stock_price, color = 'red', label = 'Predicted MSFT Stock Price')
plt.title('MSFT Stock Price Prediction')
plt.xlabel('Trading Day')
plt.ylabel('MSFT Stock Price')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>TESTING THE MODEL<jupyter_code>x_train, X_test, y_train, y_test= train_test_split(x_train, y_train, test_size=0.25, random_state=seed)
from keras import Sequential
from keras.layers import Embedding, LSTM, Dense, Dropout

embedding_size = 32

#save the model
regressor.save('saved_models.h5')
#create dummies or random variables
y_test_dummies = pd.get_dummies(y_test).values
print('Shape of Label tensor:', y_test_dummies.shape)
#model evaluation
from keras.models import load_model

regressor = load_model('saved_models.h5')
score = regressor.evaluate(X_test, y_test_dummies)

LSTM_accuracy = score[1]*100

print('Test accuracy:', score[1]*100, '%')<jupyter_output>8/8 [==============================] - 16s 69ms/step - loss: 0.0175 - accuracy: 0.9957
Test accuracy: 99.56709146499634 %
{'fraction_non_alphanumeric': 0.14354493580599145, 'fraction_numerical': 0.045203281027104136, 'mean_word_length': 4.3772770853307765, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '317986', 'license': 'MIT', 'n_tokens_mistral': 4058, 'n_tokens_neox': 3605, 'n_words': 974, 'path': 'LSTM/LSTM_model.ipynb', 'repo_name': 'patmbee/ml_project_2', 'size': '11216'}