<jupyter_start><jupyter_text>Stock Price PredictionIn this notebook, we demonstrate a reference use case where we use historical stock price data to predict the future price. The dataset we use is the daily stock price of S&P500 stocks during 2013-2018 ([data source](https://www.kaggle.com/camnugent/sandp500/)). We demostrate how to do univariate forecasting using the past 80% of the total days' MMM price to predict the future 20% days' daily price.Reference: https://github.com/jwkanggist/tf-keras-stock-pred  Get DataWe will use the close prices of MMM stock for our experiment. We will   1. download raw dataset and load into dataframe.   2. Extract the close prices of MMM stock from the dataframe into a numpy array<jupyter_code>import numpy as np
import pandas as pd
import os
# S&P 500
FILE_NAME    = 'all_stocks_5yr.csv'
SOURCE_URL   = 'https://github.com/CNuge/kaggle-code/raw/master/stock_data/' 

filepath = './data/'+ FILE_NAME
filepath = os.path.join('data', FILE_NAME)
print(filepath)
# download data
!if ! [ -d "data" ]; then mkdir data; cd data; wget https://github.com/CNuge/kaggle-code/raw/master/stock_data/individual_stocks_5yr.zip; wget https://raw.githubusercontent.com/CNuge/kaggle-code/master/stock_data/merge.sh; chmod +x merge.sh; unzip individual_stocks_5yr.zip; ./merge.sh; fi

# read data
data = pd.read_csv(filepath)
print(data[:10])

target_rows = data[data['Name']=='MMM']
print(target_rows[:10])
# extract close value
close_val = target_rows[['close']].values
print(close_val[:10])
# Visualize data

import matplotlib.pyplot as plt
plt.plot(close_val, color='blue', label='MMM daily price Raw')
plt.xlabel("Time Period")
plt.ylabel("Stock Price")
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Data Pre-processingNow we need to do data cleaning and preprocessing on the raw data. Note that this part could vary for different dataset.For the stock price data we're using, the processing contains 2 parts:1.   Data normalization such that the normalized stock prices fall in the range of 0 to 12.   Extract time series of given window sizeWe generate a built-in TSDataset to complete the whole processing.<jupyter_code>from zoo.chronos.data import TSDataset
from sklearn.preprocessing import MinMaxScaler

df = target_rows[['date', 'close']]
tsdata_train, _, tsdata_test = TSDataset.from_pandas(df, dt_col="date", target_col="close", with_split=True, test_ratio=0.2)
minmax_scaler = MinMaxScaler()

for tsdata in [tsdata_train, tsdata_test]:
    tsdata.scale(minmax_scaler, fit=(tsdata is tsdata_train))\
          .roll(lookback=50, horizon=1)

X_train, y_train = tsdata_train.to_numpy()
X_test, y_test = tsdata_test.to_numpy()
y_train, y_test = y_train[:, 0, :], y_test[:, 0, :]
X_train.shape, y_train.shape, X_test.shape, y_test.shape<jupyter_output><empty_output><jupyter_text>Time series forecastingWe use LSTMForecaster for forecasting.<jupyter_code>from zoo.chronos.model.forecast.lstm_forecaster import LSTMForecaster<jupyter_output>WARNING:tensorflow:From /home/yuyin/analytics-zoo/pyzoo/zoo/tfpark/zoo_optimizer.py:73: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.<jupyter_text>First we initiate a LSTMForecaster.*   `feature_dim` should match the dimension of the input data, so we just use the last dimension of train input data shape*   `target_dim` equals the dimension of the output data, here we set `target_dim=1` for univariate forecasting.<jupyter_code># Hyperparameters
feature_dim = X_train.shape[-1]
target_dim = 1
hidden_dim = 10
learning_rate = 0.01
batch_size = 16
epochs = 50
# build model
lstm_config = {"lstm_units": [hidden_dim] * 1, "lr": learning_rate}
forecaster = LSTMForecaster(target_dim=target_dim, feature_dim=feature_dim, **lstm_config)<jupyter_output>WARNING:tensorflow:From /home/yuyin/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.<jupyter_text>Then we use fit to train the model. Wait sometime for it to finish.<jupyter_code>%%time
forecaster.fit(x=X_train, y=y_train, batch_size=batch_size, epochs=epochs, distributed=False)<jupyter_output>WARNING:tensorflow:From /home/yuyin/anaconda3/envs/env/lib/python3.6/site-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Train on 958 samples
Epoch 1/50
958/958 [==============================] - 11s 12ms/sample - loss: 0.0659 - mean_squared_error: 0.0659
Epoch 2/50
958/958 [==============================] - 12s 12ms/sample - loss: 0.0114 - mean_squared_error: 0.0114
Epoch 3/50
958/958 [==============================] - 11s 11ms/sample - loss: 0.0081 - mean_squared_error: 0.0081
Epoch 4/50
958/958 [==============================] - 11s 12ms/sample - loss: 0.0070 - mean_squared_error: 0.0070
Epoch 5/50
958/958 [==============================] - 22s 23ms/sample - loss: 0.0062 - mean_squared_error: 0.0062
Epoch 6/50
958/958 [==============================] - 19s 20ms/sample - loss: 0.0050 -[...]<jupyter_text>After training is finished. You can use the forecaster to do prediction and evaluation.<jupyter_code># make prediction
y_pred = forecaster.predict(X_test)<jupyter_output><empty_output><jupyter_text>Since we have used standard scaler to scale the input data (including the target values), we need to inverse the scaling on the predicted values too.<jupyter_code>y_pred_unscale = tsdata_test.unscale_numpy(np.expand_dims(y_pred, axis=1))[:, 0, :]
y_test_unscale = tsdata_test.unscale_numpy(np.expand_dims(y_test, axis=1))[:, 0, :]<jupyter_output><empty_output><jupyter_text>Calculate the mean square error.<jupyter_code># evaluate with mean_squared_error
from sklearn.metrics import mean_squared_error
print("mean_squared error is", mean_squared_error(y_test_unscale, y_pred_unscale))<jupyter_output>mean_squared error is 59.547528981063884<jupyter_text>Visualize the prediction.<jupyter_code># Plot predictions
plt.plot(y_test_unscale, color='blue', label="MMM daily price Raw")
plt.plot(y_pred_unscale, color='red', label="MMM daily price Predicted")
plt.xlabel("Time Period")
plt.ylabel("Normalized Stock Price")
plt.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12909118587304003, 'fraction_numerical': 0.03546963008068199, 'mean_word_length': 6.026737967914438, 'pattern_counts': {'":': 2, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '609139', 'license': 'Apache-2.0', 'n_tokens_mistral': 2392, 'n_tokens_neox': 2157, 'n_words': 597, 'path': 'pyzoo/zoo/chronos/use-case/fsi/stock_prediction.ipynb', 'repo_name': 'cabuliwallah/analytics-zoo', 'size': '6569'}