<jupyter_start><jupyter_text>IntroIn this project, we'll be working with data from the [S&P500 Index](https://en.wikipedia.org/wiki/S%26P_500_Index). We can use historical data on the price of the S&P500 Index to make predictions about future prices. Predicting whether an index will go up or down will help us forecast how the stock market as a whole will perform. Since stocks tend to correlate with how well the economy as a whole is performing, it can also help us make economic forecasts.  The DataWe'll be working with the `sphist.csv` file containing index prices. Each row in the file contains a daily record of the price of the S&P500 Index from 1950 to 2015. The columns of the dataset are: - `Date` -- The date of the record. - `Open` -- The opening price of the day (when trading starts). - `High` -- The highest trade price during the day. - `Low` -- The lowest trade price during the day. - `Close` -- The closing price for the day (when trading is finished). - `Volume` -- The number of shares traded. - `Adj Close` -- The daily closing price, adjusted retroactively to include any corporate actions. Read more here.We'll be using this dataset to develop a predictive model. We'll train the model with data from `1950-2012`, and try to make predictions from `2013-2015`.Let's start by loading necessary models and the data into a pandas DataFrame.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.model_selection import KFold
from sklearn.cluster import KMeans
from datetime import datetime

df = pd.read_csv('sphist.csv')
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Next we'll convert the `Date`column to a date:<jupyter_code>df["Date"] = pd.to_datetime(df["Date"])
df.sort_values(by='Date', ascending=True, inplace=True)
df.reset_index(drop=True, inplace=True)<jupyter_output><empty_output><jupyter_text>Generating IndicatorsDatasets taken from the stock market need to be handled differently than datasets from other sectors when it comes time to make predictions. In a normal machine learning exercise, we treat each row as independent. Stock market data is sequential, and each observation comes a day after the previous observation. Thus, the observations are not all independent, and you can't treat them as such.Here are the indicators that we'll generate for each row: - The average price from the past `5` days. - The average price for the past `30` days. - The average price for the past `365` days. - The ratio between the average price for the past 5 days and the average price for the past 365 days. - The standard deviation of the price over the past 5 days. - The standard deviation of the price over the past 365 days. - The ratio between the standard deviation for the past 5 days, and the standard deviation for the past 365 days.<jupyter_code>#Add features that could be helpful for machine learning
df['5 Days Open'] = df['Open'].rolling(center=False, window=5).mean()
df['5 Days High'] = df['High'].rolling(center=False, window=5).mean()
df['5 Days Low'] = df['Low'].rolling(center=False, window=5).mean()
df['5 Days Volume'] = df['Volume'].rolling(center=False, window=5).mean()

df['Year'] = df['Date'].apply(lambda x: x.year)

#Adding Day of week column and set it to categorical
df['DOW'] = df['Date'].apply(lambda x: x.weekday())
dow_df = pd.get_dummies(df['DOW'])
df = pd.concat([df, dow_df], axis=1)
df = df.drop(['DOW'], axis=1)
#Shift the columns by one
df['5 Days Open'] = df['5 Days Open'].shift(1)
df['5 Days High'] = df['5 Days High'].shift(1)
df['5 Days Low'] = df['5 Days Low'].shift(1)
df['5 Days Volume'] = df['5 Days Volume'].shift(1)
# df.dropna(axis=0, inplace=True)<jupyter_output><empty_output><jupyter_text>Create Test and Train DataFrames - Remove any rows from the DataFrame that fall before `1951-01-03`. - Use the dropna method to remove any rows with `NaN` values. Pass in the `axis=0` argument to drop rows. - Generate two new dataframes to use in making our algorithm:    - `train` should contain any rows in the data with a date less than `2013-01-01`.    - `test` should contain any rows with a date greater than or equal to `2013-01-01`.<jupyter_code>df = df[df['Date'] >= datetime(year=1951, month=1, day=3)]
df.dropna(axis=0)
# Create train and test dataframes
train = df[df["Date"] < datetime(year=2013, month=1, day=1)]
test = df[df["Date"] >= datetime(year=2013, month=1, day=1)]<jupyter_output><empty_output><jupyter_text>Making Predictions<jupyter_code>features = ['5 Days Open', '5 Days Volume', '5 Days High', '5 Days Low', 'Year', 0, 1, 2, 3, 4]
lr = LinearRegression()
lr.fit(train[features], train['Close'])
predictions = lr.predict(test[features])
mae = mean_absolute_error(test['Close'] ,predictions)
print(df.tail(1))
print(mae)<jupyter_output>Date         Open         High          Low        Close  \
16589 2015-12-07  2090.419922  2090.419922  2066.780029  2077.070068   

             Volume    Adj Close  5 Days Open  5 Days High   5 Days Low  \
16589  4.043820e+09  2077.070068  2081.507959  2096.058057  2066.807983   

       5 Days Volume  Year  0  1  2  3  4  
16589   4.085838e+09  2015  1  0  0  0  0  
14.60031672337951
{'fraction_non_alphanumeric': 0.10312326741822214, 'fraction_numerical': 0.05322491221585659, 'mean_word_length': 4.533742331288344, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '342912', 'license': 'Apache-2.0', 'n_tokens_mistral': 1806, 'n_tokens_neox': 1620, 'n_words': 682, 'path': '6.5 ML in Python Intermediate/GP Predicting the Stock Market.ipynb', 'repo_name': 'orspain/Dataquest', 'size': '5411'}