<jupyter_start><jupyter_text>Brazilian stock price prediction  Table of Contents1. [Project Definition](Project-Definition)2. [Analysis](Analysis)3. [Methodology](Methodology)4. [Results](Results)5. [Conclusion](Conclusion)  Part I : Project Definition Predict stocks price or having some type of advantage in market decision is a subject of great interest. However, [Efficient Markets Hypothesis](https://www.investopedia.com/terms/e/efficientmarkethypothesis.asp) says that it is not possible to beat the market because its behaviour is completely random. But a lot of hedge funds have had high gains historically. Can we predict stock prices?  Problem Statement For this project, I was interested to study more about stock market and time series. Moreover, I was eager to know which kind of features I will use and check how well I will predict stocks in a first approach. So the challenge is to build a stock price predictor that takes daily trading data over a certain date range as input, and outputs projected estimates for given query dates. The predicted target is adjusted close stock price, it is close price adjusted historically after splits, dividends and other events. These models are built for different stocks and estimate prices for several future days.The project description is available from Udacity at this [link](https://docs.google.com/document/d/1ycGeb1QYKATG6jvz74SAMqxrlek9Ed4RYrzWNhWS-0Q/pub). All data are from [Yahoo Finance](https://finance.yahoo.com/) and extracted by [yahooquery](https://pypi.org/project/yahooquery/) API. There are two broad categories of approaches used for choose stocks to buy or sell. They are fundamental analysis and technical analysis.- Fundamental analysis: Look for aspects of a company to try estimate its real value. If the price of this stock is lower than its value buy it and if it is higher than its value sell it.- Technical analysis: Just use price and volume to discovery patterns and trends to buy or sell a stock. For this project, it will follow Technical approach. It will be created some features based in theorical and practical market knowledgeonly using past adjusted close price of the specific stock and a market index. The expected solution is not a high performance model, because this problem is really complex.But understading more about this topic and finding a starting point to tackle this subject are the expected solution.  Metrics This problem will be see as a regression problem, so the main metric calculated to evalute models is [MAE](https://en.wikipedia.org/wiki/Mean_absolute_error) (mean absolute error).For each trained model, the train data set will be splited in train and test set. The test set will be composed by 25% of the newest data and train set with the rest of data. Some stock models will be compared for 4 different target predictions days with a baseground date:- Next workday of baseground date- Next week of baseground date (approximated by next 5 workdays)- Next 2 weeks of baseground date (approximated by next 10 workdays)- Next month of base ground date (approximated by next 22 workdays)After calculated MAE for test set in this train and test approach, it will be used all training period to refit model and deploy this final model. Moreover, the fitted models can use other baseground date different from last day of train set. However, for distant days from train period the performance probably will decrease. Finally, to get some insights for future works, it will be visually analysed the shape of predicted price curve aganist real price curve.  Part 2 : Analysis  Data Exploration and Visualization<jupyter_code># Function to read from yahooquery API and save it in db
load_data = LoadData('data/Stocks.db')
load_data.load()
# Read data from db in a dataframe
engine = create_engine('sqlite:///data/Stocks.db')
stocks_df = pd.read_sql_table('stocks', engine)
stocks_df.head(3)<jupyter_output><empty_output><jupyter_text>Only some tickers were read from API. If this number would be increased, processing time was increased too with no gain in project development.<jupyter_code># Stocks read from API
stocks_df.symbol.unique().tolist()<jupyter_output><empty_output><jupyter_text>^BVSP is a brazilian stock index of most valued companies. It will be a proxy for market oscilation, similiar of SP500 in EUA.<jupyter_code>stocks_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>No nulls found<jupyter_code>stocks_df.dtypes<jupyter_output><empty_output><jupyter_text>Column types are ok  Columns description: - **symbol**: Ticker of specific stock- **date**: Reference date- **close**: Close stock price in reference date- **high**: The highest stock price achived in reference date- **open**: Open stock price in reference date- **low**: The lowest stock price achived in reference date- **volume**: Volume of transactions of specific ticker in reference date - **adjclose**: Close stock price in reference date adjusted historically when events happened such as dividends, splits and price modificators.- **dividends**: Divideds payed per unit in reference date- **splits**: Modificator ratio in number of papers in reference date. Ex.: 10 units became 11, if splits were 1.1.<jupyter_code>min_date = stocks_df.date.min().date()
max_date = stocks_df.date.max().date()

print(f'The minimun date in dataset is: {min_date}')
print(f'And this data is read every run time from API, so maximun date is {max_date}.')
# An auxiliar function to make some plots
def plot_series(df, symbol, col_date, col_value, y_label):
    ''' Plot a time series.
    
    params:
    df (dataframe): Input dataframe
    symbol (string): Specific ticker for a stock in df
    col_date (string): Column name with date value
    col_value (string): Columns name with numeric value
    y_label (string): Label to y axis.
    '''
    data = df.copy()
    data.set_index(col_date, inplace=True)
    data = data[data.symbol == symbol][col_value]
    ax = data.plot()
    ax.set_title(symbol + ' '+ col_value)
    ax.set_ylabel(y_label)
plot_series(stocks_df, 'ITSA4', 'date', 'adjclose', 'Price (R$)')<jupyter_output><empty_output><jupyter_text>For ITSA4, the price range arrives R$ 14 (Brazilian currency) and we have data since 2000.<jupyter_code>plot_series(stocks_df, 'MGLU3', 'date', 'adjclose', 'Price (R$)')<jupyter_output><empty_output><jupyter_text>For MGLU3, the price range arrives R$ 27 and we have data since 2011.<jupyter_code>plot_series(stocks_df, 'EGIE3', 'date', 'adjclose', 'Price (R$)')<jupyter_output><empty_output><jupyter_text>For EGIE3, the price range arrives R$ 51 and we have data since 2002. Both stocks have different behaviors, price ranges and historic data over the time. However, they were a good invest choice up to now due to their exponential shape.<jupyter_code>plot_series(stocks_df, '^BVSP', 'date', 'adjclose', 'Market points')<jupyter_output><empty_output><jupyter_text>Market index increased a lot over the time too. For this project only symbol, date and adjclose were used, based in Technical analysis. Therefore, it was not necessary an extensive exploratory analysis for input data. The main effort will be in feature engineering.  Part 3 : Methodology  Initial setups For modeling process, it was defined arbitrarily some parameters:- Period of 01/Aug/2020 to 30/Nov/2020 as training period. This period was chosen to have future data not used in train to evaluate performance without any problems. Moreover, training period should be recent.- Tickers:     - ITSA4 - Itau holding(biggest private bank in Latin America)    - MGLU3 - Magazine Luiza (one of the biggest retail company in Brazil)    - EGIE3 - private company in energy sector    - Targets:    - next workday    - next week    - next 2 weeks    - next monthObs. dataset has only days which market opened, so the approximation for 1 month is to read next 22 datapoints (workdays).- Forecast date: A ground date to be used to predict next n days according to the target option.Ex.: forecast_date set to 2020-12-01 and target set to 1_day, the prediction will be to 2020-12-02. If for the same target option forecast_date is set to 2020-12-11, the prediction will be for 2020-12-14 (next workday).<jupyter_code># Initial setups
start_date = pd.to_datetime('2020-08-01')
end_date = pd.to_datetime('2020-11-30')
train_range_days = (end_date - start_date).days

symbols = ['ITSA4', 'MGLU3', 'EGIE3']

targets_dict = {'1_day': 1,
                '1_week': 5,
                '2_weeks': 10,
                '1_month': 22}

forecast_date = pd.to_datetime('2020-12-01')

# path to save pickle models
models_path = 'models/models.pkl'<jupyter_output><empty_output><jupyter_text>Data Preprocessing Based on course "Machine Learning for Trading", <NAME> (Georgia Tech and Udacity), some technical indicators were created as features. It was used `ITSA4` ticker as example toexplain each feature built in this process. TrainModel is a class to prepare data, train models and calculate results. For more details, check docstrings.<jupyter_code>train_model = TrainModel(stocks_df, symbols, start_date, end_date, models_path, targets_dict)<jupyter_output><empty_output><jupyter_text>The method `prep_historical_date` gets all days between start_date and end_date, and add more 180 workdays before and 45 workdays after this period. These aditional days are used to be possible calculate a future target date in the end of the data series and to create features with a big past window date in beginning of the series.<jupyter_code># Prepare a dataframe with aditional days to be used in feature engineering for ITSA
itsa4, ibov = TrainModel.prep_historical_data(stocks_df, 'ITSA4', start_date, end_date)<jupyter_output><empty_output><jupyter_text>Explaining each feature  Daily return The method `calculate_n_day_return` calculate the difference of price between data reference as index in a row and n workdays before it. To calculate daily oscilation, just use parameter window=1.<jupyter_code># Calculate daily return in percentage
itsa4_daily_return = calculate_n_day_return(itsa4, window=1)
ax = itsa4_daily_return.plot()
ax.set_title("ITSA4 daily return")
_ = ax.set_ylabel("Daily return (%)")<jupyter_output><empty_output><jupyter_text>This plot shows that the biggest and lowest daily returns in ITSA4 for train period happened in the beginning of COVID-19 quantine in Brazil (end of march/2020). In these days, price changed 10% in a single day.  CAPM CAPM equations: $stockprice = \beta * marketprice + \alpha$ Beta is an indicator which decribes how a specific stock is correlated with the market.Alpha is an offset between stockprice and market after beta variation, it usually is zero.<jupyter_code># Just picking up a arbitrary window=30 as example
capm_df = calculate_capm(itsa4, ibov, window=30)

beta = capm_df.loc[forecast_date].beta
print(f'Beta for data reference {forecast_date}, using 30 points to calculate line coefs is {round(beta, 2)}')<jupyter_output>Beta for data reference 2020-12-01 00:00:00, using 30 points to calculate line coefs is 0.63<jupyter_text>Beta is 0.63. Therefore, if ibovespa index (good proxy for brazilian market increases 100% tomorrow, ITSA4 will increases 63%).  Simple Moving Average (SMA) SMA is mean average of adjclose price looking a window of days.<jupyter_code># Just picking up a arbitrary window=30 as example
ax = calculate_sma(itsa4, window=30).plot()
ax.set_title("ITSA4 SMA")
_ = ax.set_ylabel("SMA (%)")<jupyter_output><empty_output><jupyter_text>How larger the window is, more smooth the series becomes. There are not all 30 point to calculate average in the oldest days, it is the reason this curve is flat in the beginning. However, aditional 180 days before `start_date` was included in this series to be cut after features creation.   Exponential Moving Average (EMA) EMA is similar to SMA, but recent dates have more weight than old dates in the window.<jupyter_code># Just picking up a arbitrary window=30 as example
ax = calculate_ema(itsa4, window=30).plot()
ax.set_title("ITSA4 EMA")
_ = ax.set_ylabel("EMA (%)")<jupyter_output><empty_output><jupyter_text>Same consideration of SMA, but it gets faster variations in price due to greater weights in newest data.  Momentum Momentum is the ratio of `forecast_price` and some day before. It gives a feeling of price's direction trend.<jupyter_code># Just picking up a arbitrary window=7 as example
round(calculate_momentum(itsa4, window=7).loc[forecast_date], 2)<jupyter_output><empty_output><jupyter_text>In `forecast_date` momentum indicates almost no moving in prices.  Volatility Volatility is the standard deviation in a window of days. In stock market, this is associated with risk. How more volatile the market is, greater the risk.<jupyter_code>calculate_volatility(itsa4, window=30).plot()
ax.set_title("ITSA4 Volatility")
_ = ax.set_ylabel("Volatility")<jupyter_output><empty_output><jupyter_text>In this plot is possible to observe market response with high oscilation due to beginning of Covid-19's quarantine in Brazil (end of march 2020).  Defining windows parameters To create `vars_df` dataframe with calculated features were chosen some windows for each indicator based in practical knowledges:- https://www.investopedia.com/terms/s/sma.asp- https://www.investopedia.com/terms/e/ema.aspObs. consecutive days transformed to work days.Other windows are created arbitrarily and some indicator were combined. Example of combined indicator: `sma_momentum_36` - it is SMA of momentum series with windows 36 work days before `forecast_date`.Ticker (ITSA) feature is the last price that we have information for prediction, it was used as a feature too.  List of features:- SMA stock price for last 7, 14, 36, 73 and 146 workdays- Momentum stock price for last 36 workdays- SMA Momentum for last 7, 36 and 73 workdays (momentum fixed in 36)- SMA ^BVSP for last 73 workdays- Volatility stock price for last 73 workdays- Volatility ^BVSP for last 73 workdays- Beta between stock price and ^BVSP for last 73 workdays- EMA stock price for last 7, 14, 36, 73 and 146 workdays- Last stock price  Target creation For target creation, it was slid back n rows of `adjclose price` in ordered date time series according target option:- `1_day` - slide 1 row- `1_week` - slide 5 rows- `2_weeks` - slide 10 rows- `1_month` - slide 22 rows   Final dataframe with features and targets `prep_modeling_data` receives clean adjclose price from itsa4 and ibov and create all features that we need. If `target_flag` is True, it will create targets too.<jupyter_code># Encapsulate function to create features dataframe and drop aditional period apart from start and end date range
itsa4_vars_df = TrainModel.prep_modeling_data(itsa4, ibov, targets_dict, start_date, end_date, target_flag=True)
itsa4_vars_df.head()<jupyter_output><empty_output><jupyter_text>Implementation They are tested 3 types of models: Linear regression, random forest and neural net. Both using sklearn package. Standard scaler was used as default to normalize all features, but in random forest it does not have any effect. pipeline structure to keep it easy and organized.  Models<jupyter_code>def tunning_params(vars_df, rsearch):
    '''
    
    
    '''
    tunning_results = dict()

    for target_period in ['1_day', '1_week', '2_weeks', '1_month']:
        X, y, X_train, X_test, y_train, y_test = TrainModel.train_test_split(
                            vars_df,
                            target_period,
                            test_size_ratio=0.25)

        rsearch.fit(X_train, y_train)
        tunning_results[target_period] = [round(rsearch.cv_results_['mean_train_score'].mean() * (-1), 2),
                               round(rsearch.cv_results_['mean_test_score'].mean() * (-1), 2),
                               rsearch.best_estimator_]

    return tunning_results<jupyter_output><empty_output><jupyter_text>Linear Regression Linear regression was used as a baseline to start improving the solution. This model does not have specific params for each target period.<jupyter_code>lr = LinearRegression()
base_pipeline_lr = make_pipeline(StandardScaler(), lr)
manual_pipeline_lr = defaultdict(lambda: base_pipeline_lr)

# Just a copy of manual_lr
pipeline_lr = manual_pipeline_lr.copy()<jupyter_output><empty_output><jupyter_text>Random Forest Random Forest was set up with few estimatores. It was tested manually parameters changed, and MAE does not decrease if is increased the number of estimators.<jupyter_code>manual_rf = RandomForestRegressor(random_state=0, criterion = 'mae', n_estimators=1000)

# Any target period request returns same model setup
manual_pipeline_rf = defaultdict(lambda: make_pipeline(StandardScaler(), rf))
rf = RandomForestRegressor(random_state=0, criterion = 'mae')
base_pipeline_rf = make_pipeline(StandardScaler(), rf)
param_search_rf = {'randomforestregressor__n_estimators': range(100, 2000, 100),
           'randomforestregressor__max_depth': [3, 4, 5, 6, 8 ,10, 12, 15] + [None],
           'randomforestregressor__max_features': ['auto', 'sqrt'],
           'randomforestregressor__min_samples_leaf': [1, 2, 4],
           'randomforestregressor__min_samples_split': [2, 5, 10]
            }

tscv = TimeSeriesSplit(n_splits=2)

rsearch_rf = RandomizedSearchCV(n_iter=100,
    estimator=base_pipeline_rf, cv=tscv,
    param_distributions=param_search_rf, scoring='neg_mean_absolute_error',
    random_state=0, return_train_score=True)
## This cell takes some time, it is random search for random forest
# tunning_rf = tunning_params(itsa4_vars_df, rsearch_rf)
# tunning_rf
pipeline_rf = {
'1_day': make_pipeline(StandardScaler(),
    RandomForestRegressor(criterion='mae', max_depth=5,
                                         min_samples_leaf=2, n_estimators=500,
                                         random_state=0)),
 
'1_week': make_pipeline(StandardScaler(),
     RandomForestRegressor(criterion='mae', max_depth=4,
                                         max_features='sqrt', min_samples_leaf=2,
                                         random_state=0)),
 
'2_weeks': make_pipeline(StandardScaler(),
    RandomForestRegressor(criterion='mae', max_depth=10,
                                         n_estimators=1700, random_state=0)),
 
'1_month': make_pipeline(StandardScaler(),
    RandomForestRegressor(criterion='mae', max_depth=5,
                                         min_samples_leaf=2, min_samples_split=10,
                                         random_state=0))
}<jupyter_output><empty_output><jupyter_text>Neural Net Neural net was used with a small architecture, some variations were tested manually. If it is increased network architecture, it starts to overfit. `early_stopping` was set to True to help avoid overfitting. It had a good performance in specific cases.<jupyter_code>manual_neural_net = MLPRegressor(hidden_layer_sizes=(50, 50), activation='tanh', solver='adam',
    random_state=0, alpha=10, learning_rate='constant', max_iter=10**5,
    shuffle=False, early_stopping=True)

# Any target period request returns same model setup
manual_pipeline_neuralnet = defaultdict(lambda: make_pipeline(StandardScaler(), manual_neural_net))
neural_net = MLPRegressor(random_state=0, shuffle=False, early_stopping=True, max_iter=10**5)
base_pipeline_neuralnet = make_pipeline(StandardScaler(), neural_net)
param_search_neuralnet = {'mlpregressor__hidden_layer_sizes':[(10), (50), (100), (200),
                                                               (50, 50), (100, 100), (200, 200)],
'mlpregressor__activation': ['tanh','relu'],
'mlpregressor__solver': ['lbfgs', 'sgd', 'adam'],
'mlpregressor__learning_rate': ['constant', 'invscaling', 'adaptive'],
'mlpregressor__alpha': [0.01, 0.1, 1, 10, 100]                    
}

tscv = TimeSeriesSplit(n_splits=2)

rsearch_neuralnet = RandomizedSearchCV(n_iter=100,
    estimator=base_pipeline_neuralnet, cv=tscv, scoring='neg_mean_absolute_error',
    param_distributions=param_search_neuralnet, 
    random_state=0, return_train_score=True, verbose=3, n_jobs=5)
## This cell takes some time, it is random search for neural net
# tunning_neuralnet = tunning_params(itsa4_vars_df, rsearch_neuralnet)
# tunning_neuralnet
pipeline_neuralnet = {
'1_day': make_pipeline(StandardScaler(),
    MLPRegressor(alpha=1, early_stopping=True,
    hidden_layer_sizes=(200, 200),
    learning_rate='adaptive', max_iter=100000,
    random_state=0, shuffle=False, solver='lbfgs')),

'1_week': make_pipeline(StandardScaler(),
    MLPRegressor(activation='tanh', alpha=100, early_stopping=True,
    hidden_layer_sizes=50, max_iter=100000,
    random_state=0, shuffle=False)),

'2_weeks': make_pipeline(StandardScaler(),
    MLPRegressor(activation='tanh', alpha=0.1, early_stopping=True,
    hidden_layer_sizes=50, learning_rate='invscaling',
    max_iter=100000, random_state=0, shuffle=False,
    solver='lbfgs')),

'1_month': make_pipeline(StandardScaler(), 
    MLPRegressor(activation='tanh', alpha=10, early_stopping=True,
    hidden_layer_sizes=(50, 50), max_iter=100000,
    random_state=0, shuffle=False, solver='lbfgs'))
}
# encapsulating pipelines for evaluation with hyper param tunning by random search 
pipelines = {'lr': pipeline_lr,
             'rf': pipeline_rf,
             'neural_net': pipeline_neuralnet}
# encapsulating pipelines for evaluation with hyper param tunning manualy
manual_pipelines = {'lr': manual_pipeline_lr,
             'rf': manual_pipeline_rf,
             'neural_net': manual_pipeline_neuralnet}<jupyter_output><empty_output><jupyter_text>Refinement  DataprepThe process of building new features were long and iterative. It was necessary to go back to theoretical concepts, review function implementations and evalute results a few times. There are a lot of python package for diverse uses. However, in this case due to window days context, it was necessary to code these functions. Modeling initial solutionFor modeling step, the initial solution was linear regression but it does not fit well in time series with a lot of oscillation. It showed a lot of overfitting in test evaluations. Modeling intermediate solutionIn an intermediate solution, it was tested random forest and artificial neural network. For hyperparameter tuning, it was fixed target in `1_month` for ITSA4 stock and iterated params manualy. The premise is choosing a robust model configuration that will perform well after being properly trained (specific target and historical data), for different periods and stocks.For random forest, it was focused in:- n_estimators - it is the number of trees to vote in final prediction. If this parameter increases, it may increase model complexity, and so overfit risk. The error did not change after 1000 estimators, therefore it was chosen n_estimators equals 1000 to keep model complexity as low as possible with same error.- max_depth - this is another param to control model complexity. It sets a maximum number of division in each tree. The tests did not show an improvement in changing this param from None.For Neural network, `shuffle`, `max_iter`, `early_stopping` and `learning_rate` params were kept fixed and others were changed to test. It is not desirable that data be shuffled because in a time series problem, it would cause likely leakage of future data. `max_iter` is max iterations to fit model. If until then it does not converge, the training is stopped. And `early_stopping` is the process of stopping training if model starts to overfit (MAE test starts to increase). Finally, `learning_rate` can choose if iterations steps change over time or keep constant, for simplification it was kept constant.The parameters that have been varied are:- hidden_layer_sizes - network architecture (number of neurons and layers). This params is difficult to know. Small architecture were tested with 1 or 2 hidden layers and less than 200 neurons in each layer.- activation - activation function in internal layers. It was choosen hyperbolic tangent because logistic is not so good for it. In logistic function some values are flattened to zero. Therefore, `tanh` was picked due to popular use of data scientists. - solver - the right optimizer increases chances the model to converge and the speed up the process. `Adam` is really popular and it was arbitrarily choosen.- alpha - the l2 regularization helps to avoid overfitting. If it increases, it creates more constraints for model parameters possibilities and keeps model complexity lower. It was tested values with big steps such as 0.01, 0.1, 1, 10, 100, because small changes in this parameter does not make much effect. Modeling final solutionAs an improvement of intermediate solution, it was set one model per target period. So now, they are 4 models to find best hyperparms. Using ITSA4, it was used a random search technique which randomly chooses n hyperparam configurations to test from stipulated ranges. In random search cross-validation, it was used `TimeSeriesSplit`, it only uses data that came before validation set to avoid data leakage.For random forest, params ranges are:- n_estimators: range(100, 2000, 100). This param was tested with a small step.- max_depth: [3, 4, 5, 6, 8 ,10, 12, 15] + [None]. This param was tested with more options.- max_features': ['auto', 'sqrt']. This parameter limits number of features used to build trees. In `vars_df` there are many features correlated because it was only changed window param to create them. Using `sqrt` option can help to fix this effect.- min_samples_leaf': [1, 2, 4]. It is a constraint to let at least n samples in a leaf. It helps to avoid overtif because it does not let model becomes too specific.- min_samples_split: [2, 5, 10] It helps to avoid overfit too. This params only allows a node to split up if it has at least n samples.For neural network, params ranges are:- hidden_layer_sizes:[(10), (50), (100), (200), (50, 50), (100, 100), (200, 200)] This param was tested with more options.- activation: ['tanh','relu']. `Relu` was included in possibilities. It has a popular use too.- solver: ['lbfgs', 'sgd', 'adam']. Other types of otimizer were included. `sgd` is the basic option. There is a note in [sklearn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html) that `adam` works well for large datasets and for small datasets `lbfgs` can be better, so `lbfgs` was included too.- learning_rate: ['constant', 'invscaling', 'adaptive']. In addition to the constant parameter, it was tested news learning rate options. `invscaling` keeps decreasing learning rate over time. And `adaptive` keeps learning rate constant up to it stops to decrease training error, then it decreases learning rate.  - alpha: [0.01, 0.1, 1, 10, 100]. Same values were tested.      Part 4 : Results  Model Evaluation and Validation  ITSA4 - Hyper param manual config<jupyter_code># Calculate MAE result of test set for ITSA4 using manual param config
itsa4_manual_result = TrainModel.calculate_results(itsa4_vars_df, targets_dict, manual_pipelines)
itsa4_manual_result
# Calculate mean of each model family for manual setup
itsa4_manual_comp = itsa4_manual_result.mean(axis=1).rename('manual_conf')<jupyter_output><empty_output><jupyter_text>- Hyper param Random search config<jupyter_code># Calculate MAE result of test set for ITSA4 using random search param config
itsa4_result = TrainModel.calculate_results(itsa4_vars_df, targets_dict, pipelines)
itsa4_result
# Calculate mean of each model family for random search setup
itsa4_comp = itsa4_result.mean(axis=1).rename('random_search_conf')<jupyter_output><empty_output><jupyter_text>- Result comparison<jupyter_code># Mean of MAE test for all target periods
round(pd.concat([itsa4_manual_comp, itsa4_comp], axis=1), 2)<jupyter_output><empty_output><jupyter_text>First, for ITSA4 linear regression as first solution had a MAE test error of 1.74.Second, after implementing intermediate solution random forest achived MAE of 1.25(28% lower) and neural net 0.88(50% lower).Finally, after running random search for both models, random forest achived MAE of 1.21(5% lower than last solution) and neural net 0.81(8% lower than last solution). For ITSA4, neural net with random search optimization had a better performance and best improvement was changing model in intermediate step.  MGLU3<jupyter_code># Prep features for MGLU3
mglu3, ibov = TrainModel.prep_historical_data(stocks_df, 'MGLU3', start_date, end_date)
vars_mglu3_df = TrainModel.prep_modeling_data(mglu3, ibov, targets_dict, start_date, end_date, True)<jupyter_output><empty_output><jupyter_text>- Hyper param manual config<jupyter_code># Calculate MAE result of test set for MGLU3 using manual param config
mglu3_manual_result = TrainModel.calculate_results(vars_mglu3_df, targets_dict, manual_pipelines)
mglu3_manual_result
# Calculate mean of each model family for manual setup
mglu3_manual_comp = mglu3_manual_result.mean(axis=1).rename('manual_conf')<jupyter_output><empty_output><jupyter_text>- Hyper param Random search config<jupyter_code># Calculate MAE result of test set for MGLU3 using random search param config
mglu3_result = TrainModel.calculate_results(vars_mglu3_df, targets_dict, pipelines)
mglu3_result
# Calculate mean of each model family for random search setup
mglu3_comp = mglu3_result.mean(axis=1).rename('random_search_conf')<jupyter_output><empty_output><jupyter_text>- Result comparison<jupyter_code>round(pd.concat([mglu3_manual_comp, mglu3_comp], axis=1), 2)<jupyter_output><empty_output><jupyter_text>First, For MGLU3 linear regression as first solution had a MAE test error of 20.1. It overfitted a lot.Second, after implementing intermediate solution random forest achived MAE of 0.78(96% lower) and neural net 1.75(91% lower).Finally, after running random search for both models, random forest achived MAE of 0.7(10% lower than last solution) and neural net 6.92(74% higher than last solution). Neural net for this improvement overfitted too. For MGLU3, random forest with random search optimization had a better performance. Random search optimization showed a marginal gain compared to model change.  EGIE3<jupyter_code># Prep features for EGIE3
egie3, ibov = TrainModel.prep_historical_data(stocks_df, 'EGIE3', start_date, end_date)
vars_egie3_df = TrainModel.prep_modeling_data(egie3, ibov, targets_dict, start_date, end_date, True)<jupyter_output><empty_output><jupyter_text>- Hyper param manual config<jupyter_code># Calculate MAE result of test set for EGIE3 using manual param config
egie3_manual_result = TrainModel.calculate_results(vars_egie3_df, targets_dict, manual_pipelines)
egie3_manual_result
# Calculate mean of each model family for manual setup
egie3_manual_comp = egie3_manual_result.mean(axis=1).rename('manual_conf')<jupyter_output><empty_output><jupyter_text>- Hyper param Random search config<jupyter_code># Calculate MAE result of test set for EGIE3 using random search param config
egie3_result = TrainModel.calculate_results(vars_egie3_df, targets_dict, pipelines)
egie3_result
# Calculate mean of each model family for random search setup
egie3_comp = egie3_result.mean(axis=1).rename('random_search_conf')<jupyter_output><empty_output><jupyter_text>- Result comparison<jupyter_code>round(pd.concat([egie3_manual_comp, egie3_comp], axis=1), 2)<jupyter_output><empty_output><jupyter_text>First, for EGIE3 linear regression as first solution had a MAE test error of 6.07. It is overfitted.Second, after implementing intermediate solution random forest achived MAE of 1.19(80% lower) and neural net 13.77(127% higher). Neural net is really overfitted for this solution.Finally, after running random search for both models, random forest achived MAE of 0.7(8% lower than last solution) and neural net 2.03(85% lower than last solution). Neural net for this improvement was not soo overfitted. For EGIE3, random forest with random search optimization had a better performance. It keeps a good performance in both scenarios.  Justification Linear regression looks too simple for this type of problem, it had the worst performance in both cases. Patterns changes a lot over time in this context and it is not a robust solution.The intermediate step of testing random forest and neural net showed the best delta of improvement, even hyperparameters were tunned manualy and only a model configuration was used for both target periods. Moreover, final step of tunning a model config for each target period showed a little improvement as a fine tunning.Neural net had a really good performance for specific cases such as ITSA4. If it was tunned a specific network architecture for each stock, probably it would have a better performance. However, this point indicates neural nets perform well in restrict contexts. But for stock price prediction, in same data series can vary widely in their patterns over time. For this reason, no further investigation was done using neural networks (multilayer perceptron approach).Random forest performs well in both cases and it was chosen as the best model due to its robustness.   Perceptions abount predicted curves shape Looking for some insights, it will be analyzed visually the model performance using random forest.<jupyter_code># Prepare a test set period to plot curves
start_date_test = end_date + pd.DateOffset(days=1)
end_date_test = stocks_df.date.max() - pd.DateOffset(days=35)
print(f'In test set: \nstart_date is {start_date_test.strftime("%Y-%m-%d")}')
print(f'test_date is {end_date_test.strftime("%Y-%m-%d")}')
# Train model for all symbols and targets config params (Random Forest used)
train_model.create_models()

# Load pickle models
models = load_models(models_path)<jupyter_output><empty_output><jupyter_text>Curves shape for ITSA4<jupyter_code># Create dataframe with features and targets in test period for ITSA4
itsa4_test, ibov_test = TrainModel.prep_historical_data(stocks_df, 'ITSA4', start_date_test, end_date_test)
vars_itsa4_test_df = TrainModel.prep_modeling_data(itsa4_test, ibov_test, targets_dict, start_date_test, end_date_test, True)
# Plot result for next day prediction
ax = TrainModel.plot_test_result('1_day', models, 'ITSA4', vars_itsa4_test_df)<jupyter_output><empty_output><jupyter_text>This model only keeps price in a safe mean deslocated from real price. It try to predict oscilations only in the end.<jupyter_code># Plot result for next week prediction
ax = TrainModel.plot_test_result('1_week', models, 'ITSA4', vars_itsa4_test_df)<jupyter_output><empty_output><jupyter_text>This model keeps safe in a mean and forecast price decay tendency in last variation, but there is a error offset between the two curves.<jupyter_code># Plot result for next 2 weeks prediction
ax = TrainModel.plot_test_result('2_weeks', models, 'ITSA4', vars_itsa4_test_df)<jupyter_output><empty_output><jupyter_text>Same pattern of last plot, but a bit more smoothed and it fixs offset error in the end.<jupyter_code># Plot result for next month prediction
ax = TrainModel.plot_test_result('1_month', models, 'ITSA4', vars_itsa4_test_df)<jupyter_output><empty_output><jupyter_text>This slow shows a good prediction after `2020-12-15`, bacause the area between the two curves is small.   Curves shape for MGLU3<jupyter_code>mglu3_test, ibov_test = TrainModel.prep_historical_data(stocks_df, 'MGLU3', start_date_test, end_date_test)
vars_mglu3_test_df = TrainModel.prep_modeling_data(mglu3_test, ibov_test, targets_dict, start_date_test, end_date_test, True)
# Plot result for next day prediction
ax = TrainModel.plot_test_result('1_day', models, 'MGLU3', vars_mglu3_test_df)<jupyter_output><empty_output><jupyter_text>This model is just copying past price behaviour.<jupyter_code># Plot result for next week prediction
ax = TrainModel.plot_test_result('1_week', models, 'MGLU3', vars_mglu3_test_df)<jupyter_output><empty_output><jupyter_text>This model keeps price prediction in a safe mean, and got a wrong direction in the end.<jupyter_code># Plot result for next 2 weeks prediction
ax = TrainModel.plot_test_result('2_weeks', models, 'MGLU3', vars_mglu3_test_df)<jupyter_output><empty_output><jupyter_text>This plot shows a "safe mean" model too.<jupyter_code># Plot result for next month prediction
ax = TrainModel.plot_test_result('1_month', models, 'MGLU3', vars_mglu3_test_df)<jupyter_output><empty_output><jupyter_text>It is a bit strange the first predictions, but it keeps price in a safe mean too.   Curves shape for EGIE3<jupyter_code>egie3_test, ibov_test = TrainModel.prep_historical_data(stocks_df, 'EGIE3', start_date_test, end_date_test)
vars_egie3_test_df = TrainModel.prep_modeling_data(egie3_test, ibov_test, targets_dict, start_date_test, end_date_test, True)
# Plot result for next day prediction
ax = TrainModel.plot_test_result('1_day', models, 'EGIE3', vars_egie3_test_df)<jupyter_output><empty_output><jupyter_text>This model is only copying the past price behaviour.<jupyter_code># Plot result for next week prediction
ax = TrainModel.plot_test_result('1_week', models, 'EGIE3', vars_egie3_test_df)<jupyter_output><empty_output><jupyter_text>This model tries to make a real prediction. It is possible to see in the end of december, the model predicts the fall of prices but there is a offset error.<jupyter_code># Plot result for next 2 weeks prediction
ax = TrainModel.plot_test_result('2_weeks', models, 'EGIE3', vars_egie3_test_df)<jupyter_output><empty_output><jupyter_text>This plot makes some good predictions in price direction, for the last period it was right in price.<jupyter_code># Plot result for next month prediction
ax = TrainModel.plot_test_result('1_month', models, 'EGIE3', vars_egie3_test_df)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07486616844225367, 'fraction_numerical': 0.02075051677532199, 'mean_word_length': 5.289166666666667, 'pattern_counts': {'":': 0, '<': 185, '<?xml version=': 0, '>': 185, 'https://': 8, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '46', 'dir': 'jupyter-structured-clean-dedup', 'id': '637875', 'license': 'MIT', 'n_tokens_mistral': 11477, 'n_tokens_neox': 10703, 'n_words': 4308, 'path': 'project_report.ipynb', 'repo_name': 'felipery03/stock-prices-predictor', 'size': '37740'}