<jupyter_start><jupyter_text>Data ProcessingFirst we import all the packages we will need<jupyter_code>##import packages here
import pandas as pd
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import FunctionTransformer
import re

# We'll use this later
from numpy import meshgrid

## This sets the plot style
## to have a grid on a white background
import seaborn as sns
sns.set_style("whitegrid")<jupyter_output><empty_output><jupyter_text>Now we put various datasets we obtained into dataframesAbove each code snippet is a brief description of the dataset we are processing.  NASDAQ_dfThis dataframe contains historical data for the NASDAQ.The columns are: Date, Open, High, Low, Close, Adj Close, Volume.The dates range from Feb 5, 1971 to May 7 2021. The data is recorded daily.I do not know where we got this dataset.<jupyter_code>##import prices from ^IXIC.csv
##has data since 2/5/71
NASDAQ_df = pd.read_csv('../data/NASDAQ-all-time-daily.csv')

##drop NaN rows
NASDAQ_df.dropna()
##no need to drop more rows

##convert the dates to datetime
NASDAQ_df['Date'] = pd.to_datetime(NASDAQ_df['Date'])

NASDAQ_df = NASDAQ_df.set_index('Date')
NASDAQ_df<jupyter_output><empty_output><jupyter_text>SP500_dfThis dataframe contains historical data for the S&P 500 index.The columns are: Date, Open, High, Low, Close.The dates range from Jan 3, 1978 to May 14, 2021. The data is recorded daily.I downloaded this dataset from the Wall Street Journal.<jupyter_code>##import SP500 open/high/low/close
##has data from 1950 to 2015, add data from 2015-2021?
SP500_df = pd.read_csv('../data/sp-HistoricalPrices-wsj.csv')
#, names=['Date', 'Open', 'High', 'Low', 'Close', 'Adj Close', 'Volume']
SP500_df

##didn't need to drop any rows

##change the date column to be in datetime
SP500_df['Date'] = pd.to_datetime(SP500_df['Date'])

##set the index to be the date column
SP500_df = SP500_df.set_index('Date')

SP500_df<jupyter_output><empty_output><jupyter_text>pe_dfThis dataframe contains historical price-to-earnings ratio for the S&P 500.The columns are: date, value.The dates range from Dec 1927 to Dec 2020. The data is recorded monthly.I do not know where we got this dataset. daily dataWe added a column called daily_value, and rows for each day. The daily_value is equal to the value in the month the date occurs in.<jupyter_code>##import pe ratio from sp-500-pe-ratio-price-to-earnings-chart.csv
##has data since 1928 for each MONTH
file_pe = open('../data/sp-500-pe-ratio-price-to-earnings-chart.csv')
pe_df = pd.read_csv(file_pe, names=['date', 'value'], skiprows=[0,1,2,3,4,5,6,7])
file_pe.close()

##drop NaN rows
pe_df.dropna()
##drop header rows
pe_df = pe_df[3:]

##change date column to be in datetime format
pe_df['date'] = pd.to_datetime(pe_df['date'])

##set the index column to be the date column
pe_df = pe_df.set_index('date')

pe_df<jupyter_output><empty_output><jupyter_text>bond_dfThis dataset contains the historical bond yields for US treasury bonds.The columns are: Date, Value.The dates range from Jan 1, 1962 to May 6, 2021. The data is recorded daily.I do not know where we got this dataset.<jupyter_code>##import bond yield from 10-year-treasury-bond-rate-yield-chart.csv
##has data since 1962
file_bond = open('../data/10-year-treasury-bond-rate-yield-chart.csv')
bond_df = pd.read_csv(file_bond, names=['date', 'value'])
file_bond.close()

##drop NaN rows
bond_df = bond_df.dropna()
##drop header rows
bond_df = bond_df[1:]

##change the date column to be in datetime format
bond_df['date'] = pd.to_datetime(bond_df['date'])

##make the date column the index
bond_df = bond_df.set_index('date')

bond_df<jupyter_output><empty_output><jupyter_text>ffund_dffederal funding rate dataset<jupyter_code>##import the dataset to a csv
ffund_df = pd.read_csv('../data/fedfundrate.csv')

##convert the dates to datetime
ffund_df['DATE'] = pd.to_datetime(ffund_df['DATE'])

##make the dates the index column
ffund_df = ffund_df.set_index('DATE')

ffund_df<jupyter_output><empty_output><jupyter_text>ftass_dfthe dataset with fed total assets<jupyter_code>##import the dataset to a dataframe
ftass_df = pd.read_csv('../data/fedtotalassets.csv')

##convert the dates to datetime
ftass_df['DATE'] = pd.to_datetime(ftass_df['DATE'])

##make the date the index column
ftass_df = ftass_df.set_index('DATE')

ftass_df
##get the dates that occur in every dataset
sp500_dates = SP500_df.index
pe_dates = pd.date_range(pe_df.index[0], pe_df.index[-1] + pd.Timedelta(weeks=4,days=2))
bond_dates = bond_df.index
ffund_dates = pd.date_range(ffund_df.index[0], ffund_df.index[-1] + pd.Timedelta(weeks=4,days=1))

##get the dates that occur in every dataset
total_dates = pe_dates & sp500_dates & bond_dates & ffund_dates
total_dates
##make a dataframe with all the information
total_df = SP500_df.copy()

##drop dates that don't occur in all datasets
drop_dates = [d for d in total_df.index if d not in total_dates]
total_df = total_df.drop(drop_dates)

##add the bond column
total_df['bond'] = bond_df['value']

##add the pe column
total_df['pe'] = [pe_df.loc[d.replace(day=1), 'value'] for d in total_df.index]

##add the 'fed funds' column
total_df['fed funds'] = [ffund_df.loc[d.replace(day=1), 'FEDFUNDS'] for d in total_df.index]

##add the last column


total_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11164599148305869, 'fraction_numerical': 0.026291427513423442, 'mean_word_length': 5.7272727272727275, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '532283', 'license': 'MIT', 'n_tokens_mistral': 1984, 'n_tokens_neox': 1835, 'n_words': 585, 'path': 'scratch work/preprocessing_Shelbyoriginal.ipynb', 'repo_name': 'thinkhow/Market-Prediction-with-Macroeconomics-features', 'size': '5401'}