<jupyter_start><jupyter_text>Other ML Models for JPM Pirce Prediction- **Boosting**: MSE = 0.02065684518469964- **Random Forest**: MSE = 0.020190156713037418- **Bagging**: MSE = 0.03890373222031236  **Import Data**<jupyter_code>#import packages
import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
#import JP Morgan Stock Price
jpm = pd.read_csv('C:/Users/Heidi/Desktop/MMAI Course/894 Deep Learning/Project/0.0-sh-data-JPM.csv')<jupyter_output><empty_output><jupyter_text>**Create Feature Table and Normalize Data**In addition to the original price/volume/dividend/split feature, create some momentum features to consider the impact of previous days' prices on stock prediction, given the autocorrelation nature of time series. Example- **Lag prices**: stock price in previous days- **Moving average**: the average stock price in a defined window period - **Volatility**: standard deviation of the changes stock price- **Target** is the predicted stock price, which is the t+1 price<jupyter_code>#Create Lag features to consider the stock price in previous days
jpm['lag_t-1']=jpm['Adj_Close'].shift(1)
jpm['lag_t-2']=jpm['Adj_Close'].shift(2)
jpm['lag_t-3']=jpm['Adj_Close'].shift(3)
jpm['lag_t-4']=jpm['Adj_Close'].shift(4)
jpm['lag_t-5']=jpm['Adj_Close'].shift(5)

#Create Stock Volatility data
jpm['volatility']=jpm['Adj_Close'].rolling(21).std()

#Create Moving Average features
jpm['SMA'] = jpm['Adj_Close'].rolling(50).mean() #simple Moving Average for a window of 10 days
jpm['EMA'] = jpm['Adj_Close'].ewm(span = 25, min_periods = 12).mean() #Exponential Moving Average for a window of 12 days

#Create Return feature
jpm['return']=jpm['Adj_Close'].diff(1)/jpm['Adj_Close'].shift(1)
jpm['cumsum']=jpm['return'].cumsum()

#Create stock price predicting the target (t+1) price
jpm['target']=jpm['Adj_Close'].shift(-1)


jpm.head(5)
#Create variable X contains Features
X = np.array(jpm.iloc[50:-1,1:-2])
print('Variable X Shape', X.shape)

#Create variable Y represent target
Y=jpm['target'].iloc[50:-1].values.reshape(-1,1)
print('Variable Y Shape', Y.shape)
#Data Normalization
from sklearn import preprocessing
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
#print(scaler.fit(x_array))
MinMaxScaler(copy=True, feature_range=(0, 1))
X_Nor=scaler.fit_transform(X)
Y_Nor=scaler.fit_transform(Y)
#Split Train & Test dataset
X_train=X_Nor[:1800,:]
y_train=Y_Nor[:1800,:]
X_test=X_Nor[1801:,:]
y_test=Y_Nor[1801:,:]<jupyter_output><empty_output><jupyter_text>1. **Boosting****XGBoost** is an implementation of gradient boosted decision trees designed for speed and performance. **Gradient boosting** is an approach where new models are created that predict the residuals or errors of prior models and then added together to make the final prediction. It is called gradient boosting because it uses a gradient descent algorithm to minimize the loss when adding new models.  - The **lowest RMSE** (Root Mean Square Error) under this method is  0.021091299979040603  1) Build XGBoosting Model for Logistic Regression<jupyter_code>#Import XGBoosting Packages
import xgboost as xgb
from xgboost import XGBClassifier
from xgboost import XGBRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn import linear_model, model_selection
from sklearn.metrics import mean_squared_error 
from sklearn.model_selection import GridSearchCV
#Parameter Search using Grid Search
dmatrix = xgb.DMatrix(data=X_train, label=y_train)
gbm_param_grid = {
    'nthread':[-1],
    'learning_rate': [0.0001,0.001,0.01,0.1],
    'colsample_bytree': [0.1,0.5,1],
    'n_estimators': [50,200,500],
    'subsample':[0.4,0.6,1],
    'max_depth':[3,5,8],
    'min_child_weight':[1,5,8],
    'seed':[12,123],
    'objective':['reg:logistic']
    #'reg_alpha':[1e-6], 
    #'reg_lambda':[2e-6]
}

# Instantiate the regressor: gbm
gbm = xgb.XGBRegressor()

# Perform grid search: grid_mse
logreg_mse = GridSearchCV(estimator=gbm, param_grid=gbm_param_grid,
                        scoring='neg_mean_squared_error', cv=5, verbose=1,n_jobs=-1)
logreg_mse.fit(X_train, y_train.ravel())
y_predict=logreg_mse.predict(X_test)
y_fit=logreg_mse.predict(X_train)

# Print the best parameters and lowest log loss
print("Best parameters found: ", logreg_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(logreg_mse.best_score_)))<jupyter_output>Fitting 5 folds for each of 1944 candidates, totalling 9720 fits<jupyter_text>2) Stock Price Fitting<jupyter_code>#Fit the predicting data with the training data
plt.plot(y_fit)
plt.plot
plt.xticks(range(0,jpm.shape[0],500),jpm['Date'].loc[::500])
plt.figure(figsize = (18,9))
plt.show()
#Fit the predicting data with the testing data
plt.plot(y_predict)
plt.plot(y_test)
plt.xticks(range(0,X_test.shape[0],500),jpm['Date'].loc[len(X_train)::600])
plt.figure(figsize = (18,9))
plt.show()<jupyter_output><empty_output><jupyter_text>3) Autocorrelation Measurement First calculate Xt - Xt-1 = error term, also called white noise, which use the np.diff() function- **Graph 1 - returns**: process appears to follow random walk and centered about zero - **Autocorrelation & Partial Autocorrelation Plot**: indicate no significant serial correlation. But some serial correlation in the ACF, and PACF plots around lags 8, 14, 20, 21, 24 and 30. This means that there should be better models to describe the actual price change process. - **QQ and Probability Plots**：QQ plot visually assesses whether sample data comes from a specified distribution. For this model, the shape of QQ and Probability plots indicates that the process is close to normality but with 'heavy tails'.This code was adapted from the blog http://www.seanabu.com/2016/03/22/time-series-seasonal-ARIMA-model-in-python/<jupyter_code>def tsplot(y, lags=None, figsize=(10, 8), style='bmh'):
    if not isinstance(y, pd.Series):
        y = pd.Series(y)
    with plt.style.context(style):    
        fig = plt.figure(figsize=figsize)
        #mpl.rcParams['font.family'] = 'Ubuntu Mono'
        layout = (3, 2)
        ts_ax = plt.subplot2grid(layout, (0, 0), colspan=2)
        acf_ax = plt.subplot2grid(layout, (1, 0))
        pacf_ax = plt.subplot2grid(layout, (1, 1))
        qq_ax = plt.subplot2grid(layout, (2, 0))
        pp_ax = plt.subplot2grid(layout, (2, 1))
        
        y.plot(ax=ts_ax)
        ts_ax.set_title('Time Series Analysis Plots')
        smt.graphics.plot_acf(y, lags=lags, ax=acf_ax, alpha=0.5)
        smt.graphics.plot_pacf(y, lags=lags, ax=pacf_ax, alpha=0.5)
        sm.qqplot(y, line='s', ax=qq_ax)
        qq_ax.set_title('QQ Plot')        
        scs.probplot(y, sparams=(y.mean(), y.std()), plot=pp_ax)

        plt.tight_layout()
    return 
y_predict.shape
import statsmodels.formula.api as smf
import statsmodels.tsa.api as smt
import statsmodels.api as sm
import scipy.stats as scs
from arch import arch_model

tsplot(np.diff(y_predict),30)<jupyter_output><empty_output><jupyter_text>2. **Random Forest**<jupyter_code># Import the package
from sklearn.ensemble import RandomForestRegressor

#Define paramate
rf_param_grid = {'n_estimators':[5, 100, 200], 'max_depth': [3,5,8], 'max_features': [7,15,21], 'random_state': [2,42,123]}

# Instantiate the regressor: gbm
rf = RandomForestRegressor()

# Perform grid search: grid_mse
rf_mse = GridSearchCV(estimator=rf, param_grid=rf_param_grid,
                        scoring='neg_mean_squared_error', cv=5, verbose=1,n_jobs=-1)
rf_mse.fit(X_train, y_train.ravel())
rf_test_predictions=rf_mse.predict(X_test)
rf_train_predictions=rf_mse.predict(X_train)

# Print the best parameters and lowest MSE
print("Best parameters found: ", rf_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(rf_mse.best_score_)))
plt.plot(y_train)
plt.plot(rf_train_predictions)
plt.xticks(range(0,jpm.shape[0],500),jpm['Date'].loc[::500])
plt.figure(figsize = (18,9))
plt.show()
#Fit the predicting data with the testing data
plt.plot(rf_test_predictions)
plt.plot(y_test)
plt.xticks(range(0,X_test.shape[0],200),jpm['Date'].loc[len(X_train)::200])
plt.figure(figsize = (18,9))
plt.show()
import statsmodels.formula.api as smf
import statsmodels.tsa.api as smt
import statsmodels.api as sm
import scipy.stats as scs
from arch import arch_model

tsplot(np.diff(rf_test_predictions),30)<jupyter_output><empty_output><jupyter_text>3. **Bagging**<jupyter_code>from sklearn.ensemble import BaggingRegressor

#Define paramate
bag_param_grid = {
    'n_estimators' : [50, 100, 200]
    }

# Instantiate the regressor: gbm
bag = BaggingRegressor()

# Perform grid search: grid_mse
bag_mse = GridSearchCV(estimator=bag, param_grid=bag_param_grid,
                        scoring='neg_mean_squared_error', cv=3, verbose=1,n_jobs=-1)

bag_mse.fit(X_train, y_train.ravel())
bag_test_predictions=bag_mse.predict(X_test)
bag_train_predictions=bag_mse.predict(X_train)

# Print the best parameters and lowest MSE
print("Best parameters found: ", bag_mse.best_params_)
print("Lowest RMSE found: ", np.sqrt(np.abs(bag_mse.best_score_)))
plt.plot(y_train)
plt.plot(bag_train_predictions)
plt.xticks(range(0,jpm.shape[0],500),jpm['Date'].loc[::500])
plt.figure(figsize = (18,9))
plt.show()
#Fit the predicting data with the testing data
plt.plot(bag_test_predictions)
plt.plot(y_test)
plt.xticks(range(0,X_test.shape[0],200),jpm['Date'].loc[len(X_train)::200])
plt.figure(figsize = (18,9))
plt.show()
import statsmodels.formula.api as smf
import statsmodels.tsa.api as smt
import statsmodels.api as sm
import scipy.stats as scs
from arch import arch_model

tsplot(np.diff(bag_test_predictions),30)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12874220905282519, 'fraction_numerical': 0.03617043016246041, 'mean_word_length': 5.676671214188268, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '462826', 'license': 'MIT', 'n_tokens_mistral': 3766, 'n_tokens_neox': 3464, 'n_words': 840, 'path': 'user/heidi/3.0-hz-Other ML Models Boost_Bagging_RF.ipynb', 'repo_name': 'ssh24/MMAI-894-Project', 'size': '9787'}