<jupyter_start><jupyter_text>Capstone ProjectIn this Capstone Project we will be engaging with the stock price data for several hundred stocks over 5 years, which can be found at:(./Data/prices-split-adjusted.csv.zip).  The data is from [Kaggle](https://www.kaggle.com/dgawlik/nyseprices-split-adjusted.csv) and is available in its original form there under a CC0 license. We will be using a slightly preprocessed version in the repo.  In this notebook we will be looking at financial forecasting with machine learning.  This is inherently one of the hardest problems in machine learning, because some of the most advanced and well-funded technical teams in the world are trying to use machine learning and other techniques to find patterns in the financial data.  When they find patterns and if they trade on those findings, prices will move in a way that makes those patterns less pronounced over time. This is not to say that this is not a fun and rewarding area.  Just do not get discouraged if you don't find an instant money machine.   Outline:0. Background1. Preparing our tools2. Importing and describing the data.3. Exploring, cleaning and visualizing the data3. Developing analytics4. Preparing and splitting our data5. Building our first model6. Extending to other ML models7. Ideas for further strategies8. Wrapping up OptionsAs we progress you are encouraged to take this dataset further. You are also encouraged to explore any aspects of the data. Develop your own algorithms. Be explicit about your inquiry and success in predicting effects on our world. Warning: Not financial adviceThis exercise is meant purely for educational purposes, uses many simplifications and is not intended, nor should be considered as financial advice. There are many risks involved in implementation of financial trading strategies that are not considered nor described here.  Setting upIf you have not yet set up your environment, you can easily do so with VS Code, and the python extension and Anaconda.  For VSCode go here: [https://code.visualstudio.com/]and then you can follow these instructions:[https://code.visualstudio.com/docs/python/data-science-tutorial]0. BackgroundMachine learning is of increasing importance in finance. As volumes of data grow ever faster, the need for machine driven models to find patterns in that data becomes ever more important.  In the ever-accelerating race to better process data into predictions about securities prices, machine learning has become an important tool, because it is good at finding patterns in large amounts of data.  Today we will be examining patterns in stock prices themselves to practice developing models to predict future set prices. If there are systematic trends, patterns or reversals, then we may detect them. While the chances that we discover totally new and unexploited price patterns today is low, we will practice organizing our data, creating and analyzing machine learning models that will give us the tools to develop state of the art signals of value.Goals: 1. Become familiar and practice the process of building machine learning models as they relate to financial data.  2. Understand the special processing that is required when working with time series data such those found in finance.  1. Preparing our tools.Let's review our standard imports:- numpy for rapid numerical calculations with fast vectorized C implementations- pandas for processing data- matplotlib and Seaborn for visualizing charts- scikit-learn (imported as sklearn) is the de facto standard machine learning library in the pydata ecosystem.   Additionally, we will be using [pandas_profiling](https://github.com/pandas-profiling/pandas-profiling) which is a newer convenience package that helps by putting together much of our initial *boilerplate* exploratory data analysis code.<jupyter_code># Bring our tools in:
import numpy             as np
import pandas            as pd
import matplotlib.pyplot as plt
import seaborn as sns

from pandas_profiling import ProfileReport
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor, ExtraTreesRegressor
from sklearn.model_selection import TimeSeriesSplit

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Importing and describing the dataNow we are ready to import our data.  The value rows can be set if you only want to import a small subset due to computer memory or speed constraints.<jupyter_code># We are using information from the data source description to know that date is a column containing just what is says
rows = None
stocks = pd.read_csv('./Data/prices-split-adjusted.csv.zip', nrows=rows, parse_dates=['date'], index_col='date')<jupyter_output><empty_output><jupyter_text>Now that we have our data successfully loaded, let's explore what we have. First, summarize the dataframe via the info method to validate the data reading and parsing.  When looking at the info report, it is best practice to note that each column is the expected type, noting that strings are reported as object.  Also note if there are null values, how many values there are and what the columns are.  We do not have a data dictionary in this case.  But if you were lucky enough to have access to a data dictionary, this is a good time to check that the dictionary matches what you actually have.  Discrepancies could be the result of mis-parsing, undocumented schema changes, documentation that is not up to date, or a number of other reasons.<jupyter_code>stocks.info() # Look at the descriptions of the columns<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 851264 entries, 2016-01-05 to 2016-12-30
Data columns (total 6 columns):
 #   Column  Non-Null Count   Dtype  
---  ------  --------------   -----  
 0   symbol  851264 non-null  object 
 1   open    851264 non-null  float64
 2   close   851264 non-null  float64
 3   low     851264 non-null  float64
 4   high    851264 non-null  float64
 5   volume  851264 non-null  float64
dtypes: float64(5), object(1)
memory usage: 45.5+ MB<jupyter_text>Everything looks as expected in info.  The string column symbol is the trading symbol, also known in finance as the ticker.  The dates were parsed as expected and all the other columns are numeric.  Now we can look at the first few rows of data to get a sample view.<jupyter_code>stocks.head(10)<jupyter_output><empty_output><jupyter_text>While the head gives a good preview of a piece of the data, it may not be a great overall view of the entire dataset, especially for larger data sets or ones that may have been sorted at some point.  However, we are more comfortable that the dates were parsed correctly.  We can investigate numerical columns with describe.<jupyter_code>stocks.describe()<jupyter_output><empty_output><jupyter_text>Pandas ProfilingData exploration can start with a turnkey tool like pandas-profiling.  The important this with this is to make sure you actually look at the report and digest the output.  Make it the start of your investigation.<jupyter_code># Minimal avoids expensive calculations that won't have much insight for us and are slow.
profile = ProfileReport(stocks, minimal=True)
profile.to_widgets()<jupyter_output><empty_output><jupyter_text>Exercise: Examine this report and see what insights you notice.  What do you notice about the data?  Are there ways to slice the data that would give more information?<jupyter_code># Blank cell left for student exploration<jupyter_output><empty_output><jupyter_text>Looking at an individual stock gives a much clearer impression of the distributions of each column.  Even if you can not do this for every stock, taking a sample can be very helpful.    2. Exploring, cleaning and visualizing the data Modeling our dataOne of the most important steps in preparing your data is to think of how we want to think about it.  For those who have worked with SQL, this is like identifying what the key to your table is. For this data exploration, we can think of the index into the data as being a a compound key of both the symbol (the ticker) and the date.  We will process the data in a way that will make a time series model for each stock.  Predicting the future based on what has happened in the past for that individual stock.  Thought Question: Can you think of other ways that you might want to consider this data?  What questions might prompt you to think of this data having a different data model?  The data model is not fixed but is a lens that lets us look at our data.    Visualizing our dataIt is often helpful to look at our data visually to see if there are any issues that "look funny."  With experience, just looking at the data can help us understand it in a short amount of time.  This is often the step where domain expertise (in this case the financial markets) is especially useful.  Be sure to look at the histograms of the report above to see if they make sense to you.  Exercise: Are there other visualizations that would give you greater understanding?<jupyter_code># left blank for student answer<jupyter_output><empty_output><jupyter_text>3. Feature engineeringFor modelling data with machine learning, it is helpful to transform the data into a form that is closer to the theoretical expectations where the ML models should perform well. Let's transform the data into returns and generate other features.  We will transform returns with logarithms based on financial research that log returns are closer to normally distributed and (statistically) stable. The function below is just a sample of feature transformations.  Taking the logarithms can help deal with skewed data as we saw we have in the pandas-profile report.To be honest, which variables you use and how you transform them is largely dependent on domain expertise and traditions of the field. It can also be a matter of trial and error, although that can lead to overfitting.  We will discuss overfitting a little bit later.<jupyter_code>def feature_target_generation(df):
    """
    df: a pandas dataframe containing numerical columns
    num_days_ahead: an integer that can be used to shift the prediction value from the future into a prior row.
    """

    # The following line ensures the data is in date order    
    features = pd.DataFrame(index=df.index).sort_index() 
    features['f01'] = np.log(df.close / df.open) # intra-day log return
    features['f02'] = np.log(df.open / df.close.shift(1)) # overnight log return

    features['f03'] = df.volume # try both regular and log volume
    features['f04'] = np.log(df.volume) 
    features['f05'] = df.volume.diff() # 1-day absolute change in volume
    features['f06'] = df.volume.pct_change() # 1-day relative change in volume
    # The following are rolling averages of different periods
    features['f07'] = df.volume.rolling(5, min_periods=1).mean().apply(np.log)
    features['f08'] = df.volume.rolling(10, min_periods=1).mean().apply(np.log)
    features['f09'] = df.volume.rolling(30, min_periods=1).mean().apply(np.log)

    # More of our original data: low, high and close
    features['f10'] = df.low 
    features['f11'] = df.high
    features['f12'] = df.close
    # The Intraday trading spread measures how far apart the high and low are
    features['f13'] = df.high - df.low 
    # These are log returns over different time periods 
    features['f14'] = np.log(df.close / df.close.shift(1)) # 1 day log return
    features['f15'] = np.log(df.close / df.close.shift(5)) # 5 day log return
    features['f16'] = np.log(df.close / df.close.shift(10)) # 10 day log return
    return features<jupyter_output><empty_output><jupyter_text>In Machine Learning, we need to predict something, typically called our target or predictor. Let's predict the value of the stock price 10 days into the future, using "prediction_horizon".  Here 10 is a hyperparameter that is somewhat arbitrary.  We may want to try different horizons to see if we are better at predicting the near future or the long term.  The ticker lets us start by testing on a single stock for speed in training.  We want to look at different periods of history to see how we do.  We will use overlapping sets with n_splits.<jupyter_code># Let's generate a list of tickers so we can easily select them
ticker_list = stocks.symbol.unique()

# these are hyperparameters you can play with or tune
prediction_horizon = -5 # this is a negative number by convention
ticker = 'MSFT' # choose any ticker
n_splits = 5 

# Make an individual model for each ticker/symbol
features = feature_target_generation(stocks[stocks.symbol==ticker])<jupyter_output><empty_output><jupyter_text>5. Preparing and splitting our dataIt is important that we separate our training and test data.  Since our rows are already time ordered, we can easily do splits.  This is one of the areas where times series data is different from other machine learning problems.<jupyter_code># We are trying to predict the price prediction_horizon days in the future.  So we take the future value and move it prediction_horizon into the past to line up our data in the Scikit-learn format.  
y = features.f12.shift(prediction_horizon)
# The latest (prediction_horizon) rows will have nans because we have no future data, so let's drop them.
shifted = ~np.isnan(y)
X = features[y.notna()] # Remove the rows that do not have valid target values
y = y[shifted] # Remove the rows that do not have valid target values

# Split the history into different backtesting regimes
tscv = TimeSeriesSplit(n_splits=n_splits)
print(tscv)
 # Review the features
 features.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 1762 entries, 2010-01-04 to 2016-12-30
Data columns (total 16 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   f01     1762 non-null   float64
 1   f02     1761 non-null   float64
 2   f03     1762 non-null   float64
 3   f04     1762 non-null   float64
 4   f05     1761 non-null   float64
 5   f06     1761 non-null   float64
 6   f07     1762 non-null   float64
 7   f08     1762 non-null   float64
 8   f09     1762 non-null   float64
 9   f10     1762 non-null   float64
 10  f11     1762 non-null   float64
 11  f12     1762 non-null   float64
 12  f13     1762 non-null   float64
 13  f14     1761 non-null   float64
 14  f15     1757 non-null   float64
 15  f16     1752 non-null   float64
dtypes: float64(16)
memory usage: 234.0 KB<jupyter_text>6. Building our first modelThis is a regression problem.  Why is that so?   Linear regressionIn our ML framework we can use linear regression, just as in standard statistics or econometrics.<jupyter_code>def model_ts_report(model, tscv, X, y, impute=False):
    """
    Fit the model and then run time series backtests.
    """
    # Loop through the backtests
    for train_ind, test_ind in tscv.split(X): 
        # Report on the time periods
        print(f'Train is from {X.iloc[train_ind].index.min()} to {X.iloc[train_ind].index.max()}. ')
        print(f'Test is from {X.iloc[test_ind].index.min()} to {X.iloc[test_ind].index.max()}. ')
        # Generate training and testing features and target for each fold.
        X_train, X_test = X.iloc[train_ind], X.iloc[test_ind]
        y_train, y_test = y.iloc[train_ind], y.iloc[test_ind]

        if impute==True:
            # Since linear regression cannot deal with NaN, we need to impute.  There may be the better choices.
            X_train.fillna(0, inplace=True)
            X_test.fillna(0, inplace=True)
        
        # Fit the model
        model.fit(X_train, y_train)

        # Predict and measure on the training data
        y_pred_train = model.predict(X_train) 
        print("Training results:")
        print("RMSE:", mean_squared_error(y_train, y_pred_train, squared=False))

        # Predict and measure on the testing data
        y_pred_test = model.predict(X_test)
        print("Test results:")
        print("RMSE:", mean_squared_error(y_test, y_pred_test, squared=False))
        print("")
    
from sklearn.linear_model import LinearRegression
# Fit and report on a linear model
lm = LinearRegression()
model_ts_report(lm, tscv, X, y, impute=True)<jupyter_output>Train is from 2010-01-04 00:00:00 to 2011-03-08 00:00:00. 
Test is from 2011-03-09 00:00:00 to 2012-05-03 00:00:00. 
Training results:
RMSE: 0.7646691835724355
Test results:
RMSE: 0.8845465388255741

Train is from 2010-01-04 00:00:00 to 2012-05-03 00:00:00. 
Test is from 2012-05-04 00:00:00 to 2013-07-03 00:00:00. 
Training results:
RMSE: 0.8046921042806601
Test results:
RMSE: 0.9584478611049053

Train is from 2010-01-04 00:00:00 to 2013-07-03 00:00:00. 
Test is from 2013-07-05 00:00:00 to 2014-08-29 00:00:00. 
Training results:
RMSE: 0.8388520878621472
Test results:
RMSE: 1.219972645569116

Train is from 2010-01-04 00:00:00 to 2014-08-29 00:00:00. 
Test is from 2014-09-02 00:00:00 to 2015-10-27 00:00:00. 
Training results:
RMSE: 0.9135655974444156
Test results:
RMSE: 1.857836380010567

Train is from 2010-01-04 00:00:00 to 2015-10-27 00:00:00. 
Test is from 2015-10-28 00:00:00 to 2016-12-22 00:00:00. 
Training results:
RMSE: 1.1573573014112757
Test results:
RMSE: 1.5623450539790187<jupyter_text>As we look at our results, we can see that in each period we do better in training than in testing.  That is typical of finance and machine learning more generally. Interestingly, we are able to explain 90% of the variance with our first linear model.  If you have questions about the root-mean-squared-error (RMSE), please see the Microsoft learn module on machine learning. Ensemble ModelLet's try a Random Forest, which is a commonly used model that blends a group of decision trees, each of which have access to a sub-sample of features.  It is commonly used because it it tends to work well with relatively little tuning of hyperparameters and is somewhat less likely to overfit.  This is NOT a classic model commonly used in econometrics, mainly because it does not have a linear structure that corresponds with econometric theory.  However, it is often used in predictions for items other than finance.<jupyter_code># Initiate a Random Forest
rf = RandomForestRegressor()
model_ts_report(rf, tscv, X, y, impute=True) # Report on the random forest<jupyter_output>Train is from 2010-01-04 00:00:00 to 2011-03-08 00:00:00. 
Test is from 2011-03-09 00:00:00 to 2012-05-03 00:00:00. 
Training results:
RMSE: 0.23049077731828058
Test results:
RMSE: 1.1669160674539722

Train is from 2010-01-04 00:00:00 to 2012-05-03 00:00:00. 
Test is from 2012-05-04 00:00:00 to 2013-07-03 00:00:00. 
Training results:
RMSE: 0.24843440252946497
Test results:
RMSE: 1.3551951722514546

Train is from 2010-01-04 00:00:00 to 2013-07-03 00:00:00. 
Test is from 2013-07-05 00:00:00 to 2014-08-29 00:00:00. 
Training results:
RMSE: 0.25784945977455104
Test results:
RMSE: 5.295733162857246

Train is from 2010-01-04 00:00:00 to 2014-08-29 00:00:00. 
Test is from 2014-09-02 00:00:00 to 2015-10-27 00:00:00. 
Training results:
RMSE: 0.27349224952765466
Test results:
RMSE: 2.944342872231963

Train is from 2010-01-04 00:00:00 to 2015-10-27 00:00:00. 
Test is from 2015-10-28 00:00:00 to 2016-12-22 00:00:00. 
Training results:
RMSE: 0.38488931689432365
Test results:
RMSE: 5.006480972041069[...]
{'fraction_non_alphanumeric': 0.06500541711809317, 'fraction_numerical': 0.06433472630655729, 'mean_word_length': 4.062418386001567, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '570807', 'license': 'MIT', 'n_tokens_mistral': 6386, 'n_tokens_neox': 5431, 'n_words': 2574, 'path': 'workshop-resources/data-science-and-machine-learning/Machine_Learning_2/stock-project/Capstone.ipynb', 'repo_name': 'runewake2/Reactors', 'size': '19383'}