<jupyter_start><jupyter_text>On this notebook the best models and input parameters will be searched for. The problem at hand is predicting the price of any stock symbol 56 days ahead, assuming one model for all the symbols. The best training period length, base period length, and base period step will be determined, using the MRE metrics (and/or the R^2 metrics). The step for the rolling validation will be determined taking into consideration a compromise between having enough points, and the time needed to compute the validation.<jupyter_code># Basic imports
import os
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import datetime as dt
import scipy.optimize as spo
import sys
from time import time
from sklearn.metrics import r2_score, median_absolute_error

%matplotlib inline

%pylab inline
pylab.rcParams['figure.figsize'] = (20.0, 10.0)

%load_ext autoreload
%autoreload 2

sys.path.append('../../')
import predictor.feature_extraction as fe
import utils.preprocessing as pp
import utils.misc as misc
AHEAD_DAYS = 56<jupyter_output><empty_output><jupyter_text>Let's get the data.<jupyter_code>datasets_params_list_df = pd.read_pickle('../../data/datasets_params_list_df.pkl')
print(datasets_params_list_df.shape)
datasets_params_list_df.head()
train_days_arr = 252 * np.array([1, 2, 3])
params_list_df = pd.DataFrame()

for train_days in train_days_arr:
    temp_df = datasets_params_list_df[datasets_params_list_df['ahead_days'] == AHEAD_DAYS].copy()
    temp_df['train_days'] = train_days
    params_list_df = params_list_df.append(temp_df, ignore_index=True)
print(params_list_df.shape)
params_list_df.head()<jupyter_output>(15, 9)<jupyter_text>Let's find the best params set for some different models  - Dummy Predictor (mean)<jupyter_code>tic = time()

from predictor.dummy_mean_predictor import DummyPredictor
PREDICTOR_NAME = 'dummy'

# Global variables
eval_predictor = DummyPredictor()
step_eval_days = 60  # The step to move between training/validation pairs
params = {'eval_predictor': eval_predictor, 'step_eval_days': step_eval_days}

results_df = misc.parallelize_dataframe(params_list_df, misc.apply_mean_score_eval, params)

results_df['r2'] = results_df.apply(lambda x: x['scores'][0], axis=1)
results_df['mre'] = results_df.apply(lambda x: x['scores'][1], axis=1)
# Pickle that!
results_df.to_pickle('../../data/results_ahead{}_{}_df.pkl'.format(AHEAD_DAYS, PREDICTOR_NAME))
results_df['mre'].plot()

print('Minimum MRE param set: \n {}'.format(results_df.iloc[np.argmin(results_df['mre'])]))
print('Maximum R^2 param set: \n {}'.format(results_df.iloc[np.argmax(results_df['r2'])]))

toc = time()
print('Elapsed time: {} seconds.'.format((toc-tic)))<jupyter_output>Generating: base28_ahead56_train756
Generating: base7_ahead56_train252
Generating: base112_ahead56_train252
Generating: base56_ahead56_train504
Evaluating approximately 86 training/evaluation pairs
Evaluating approximately 78 training/evaluation pairs
Evaluating approximately 81 training/evaluation pairs
Evaluating approximately 85 training/evaluation pairs
Approximately 96.3 percent complete.     (0.36465896953931198, 0.15780714555021338)
Generating: base7_ahead56_train504
Approximately 101.2 percent complete.    Evaluating approximately 82 training/evaluation pairs
Approximately 92.3 percent complete.    (0.023225128508772255, 0.11495476261730708)
Generating: base14_ahead56_train252
Approximately 1.2 percent complete.    Evaluating approximately 86 training/evaluation pairs
Approximately 3.5 percent complete.      (0.27509762291129858, 0.13358270498689478)
Generating: base112_ahead56_train504
Approximately 6.1 percent complete.     Evaluating approximately 80 training/evaluation pair[...]<jupyter_text>- Linear Predictor<jupyter_code>tic = time()

from predictor.linear_predictor import LinearPredictor
PREDICTOR_NAME = 'linear'

# Global variables
eval_predictor = LinearPredictor()
step_eval_days = 60  # The step to move between training/validation pairs
params = {'eval_predictor': eval_predictor, 'step_eval_days': step_eval_days}

results_df = misc.parallelize_dataframe(params_list_df, misc.apply_mean_score_eval, params)

results_df['r2'] = results_df.apply(lambda x: x['scores'][0], axis=1)
results_df['mre'] = results_df.apply(lambda x: x['scores'][1], axis=1)
# Pickle that!
results_df.to_pickle('../../data/results_ahead{}_{}_df.pkl'.format(AHEAD_DAYS, PREDICTOR_NAME))
results_df['mre'].plot()

print('Minimum MRE param set: \n {}'.format(results_df.iloc[np.argmin(results_df['mre'])]))
print('Maximum R^2 param set: \n {}'.format(results_df.iloc[np.argmax(results_df['r2'])]))

toc = time()
print('Elapsed time: {} seconds.'.format((toc-tic)))<jupyter_output>Generating: base56_ahead56_train504
Generating: base112_ahead56_train252
Generating: base28_ahead56_train756
Generating: base7_ahead56_train252
Evaluating approximately 86 training/evaluation pairs
Evaluating approximately 78 training/evaluation pairs
Evaluating approximately 81 training/evaluation pairs
Evaluating approximately 85 training/evaluation pairs
Approximately 96.5 percent complete.     (-0.14681918662261664, 0.12582692470388054)
Generating: base14_ahead56_train252
Approximately 91.4 percent complete.    Evaluating approximately 86 training/evaluation pairs
Approximately 95.1 percent complete.     (0.50256742402460519, 0.13553335951119391)
Generating: base7_ahead56_train504
Evaluating approximately 82 training/evaluation pairs
Approximately 97.4 percent complete.     (0.41564260359726168, 0.12150985324601241)
Generating: base112_ahead56_train504
Approximately 98.7 percent complete.    Evaluating approximately 80 training/evaluation pairs
Approximately 1.2 percent complete.  [...]<jupyter_text>- Random Forest model<jupyter_code>tic = time()

from predictor.random_forest_predictor import RandomForestPredictor
PREDICTOR_NAME = 'random_forest'

# Global variables
eval_predictor = RandomForestPredictor()
step_eval_days = 60  # The step to move between training/validation pairs
params = {'eval_predictor': eval_predictor, 'step_eval_days': step_eval_days}

results_df = misc.parallelize_dataframe(params_list_df, misc.apply_mean_score_eval, params)

results_df['r2'] = results_df.apply(lambda x: x['scores'][0], axis=1)
results_df['mre'] = results_df.apply(lambda x: x['scores'][1], axis=1)
# Pickle that!
results_df.to_pickle('../../data/results_ahead{}_{}_df.pkl'.format(AHEAD_DAYS, PREDICTOR_NAME))
results_df['mre'].plot()

print('Minimum MRE param set: \n {}'.format(results_df.iloc[np.argmin(results_df['mre'])]))
print('Maximum R^2 param set: \n {}'.format(results_df.iloc[np.argmax(results_df['r2'])]))

toc = time()
print('Elapsed time: {} seconds.'.format((toc-tic)))<jupyter_output>Generating: base7_ahead56_train252
Generating: base112_ahead56_train252
Generating: base56_ahead56_train504
Generating: base28_ahead56_train756
Evaluating approximately 86 training/evaluation pairs
Evaluating approximately 78 training/evaluation pairs
Evaluating approximately 81 training/evaluation pairs
Evaluating approximately 85 training/evaluation pairs
Approximately 101.2 percent complete.    (-0.90869295385223947, 0.13948508549601199)
Generating: base14_ahead56_train252
Approximately 9.0 percent complete.    Evaluating approximately 86 training/evaluation pairs
Approximately 101.2 percent complete.    (-0.55035695757776726, 0.13744468223625314)
Generating: base28_ahead56_train252
Evaluating approximately 86 training/evaluation pairs
Approximately 44.4 percent complete.     (-0.40420138483344326, 0.14167447669880928)
Generating: base56_ahead56_train252
Evaluating approximately 86 training/evaluation pairs
Approximately 100.0 percent complete.    (0.36524128059059779, 0.13926840080[...]
{'fraction_non_alphanumeric': 0.09915966386554621, 'fraction_numerical': 0.08312863606981254, 'mean_word_length': 6.991735537190083, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '585537', 'license': 'MIT', 'n_tokens_mistral': 2960, 'n_tokens_neox': 2461, 'n_words': 606, 'path': 'notebooks/prod/.ipynb_checkpoints/n05_day56_model_choosing_close_feat_all_syms_equal-checkpoint.ipynb', 'repo_name': 'mtasende/Machine-Learning-Nanodegree-Capstone', 'size': '7735'}