<jupyter_start><jupyter_text>Download and store dataThis notebook contains information on downloading the Quandl Wiki stock prices and a few other sources that we use throughout the book.<jupyter_code># Imports & Settings
import warnings
warnings.filterwarnings('ignore')
from pathlib import Path
import requests
from io import BytesIO
from zipfile import ZipFile, BadZipFile

import numpy as np
import pandas as pd
import pandas_datareader.data as web
from sklearn.datasets import fetch_openml

pd.set_option('display.expand_frame_repr', False)<jupyter_output><empty_output><jupyter_text>DATA_STORE = Path('assets.h5')<jupyter_code>DATA_STORE = Path('assets.h5')
df = (pd.read_csv('wiki_prices.csv',
                 parse_dates=['date'],
                 index_col=['date', 'ticker'],
                 infer_datetime_format=True)
     .sort_index())<jupyter_output><empty_output><jupyter_text>Quandl Wiki PricesQuandl makes available a dataset with stock prices, dividends and splits for 3000 US publicly-traded companies. Quandl decided to discontinue support in favor of its commercial offerings but the historical data are still useful to demonstrate the application of the machine learning solutions in the book, just ensure you implement your own algorithms on current data.```As of April 11, 2018 this data feed is no longer actively supported by the Quandl community. We will continue to host this data feed on Quandl, but we do not recommend using it for investment or analysis.```Follow the instructions to create a free Quandl) accountDownload the entire WIKI/PRICES dataExtract the .zip file,Move to this directory and rename to wiki_prices.csvRun the below code to store in fast HDF format (see Chapter 02 on Market & Fundamental Data for details).<jupyter_code>print(df.info(null_counts=True))
with pd.HDFStore(DATA_STORE) as store:
    store.put('quandl/wiki/prices', df)<jupyter_output><empty_output><jupyter_text>Wiki Prices MetadataAs of writing, the following instructions no longer work because Quandl changed its API:Follow the instructions to create a free Quandl) account if you haven't done so yetFind link to download wiki metadata under Companies](https://www.quandl.com/databases/WIKIP/documentation) or use the download link with your API_KEY: https://www.quandl.com/api/v3/databases/WIKI/metadata?api_key=Extract the .zip file,Move to this directory and rename to wiki_stocks.csvRun the following code to store in fast HDF formatInstead, load the file wiki_stocks.csv as described and store in HDF5 format.<jupyter_code>df = pd.read_csv('wiki_stocks.csv')
# no longer needed
# df = pd.concat([df.loc[:, 'code'].str.strip(),
#                 df.loc[:, 'name'].str.split('(', expand=True)[0].str.strip().to_frame('name')], axis=1)

print(df.info(null_counts=True))
with pd.HDFStore(DATA_STORE) as store:
    store.put('quandl/wiki/stocks', df)<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 3199 entries, 0 to 3198
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   code    3199 non-null   object
 1   name    3199 non-null   object
dtypes: object(2)
memory usage: 50.1+ KB
None<jupyter_text>S&P 500 PricesThe following code downloads historical S&P 500 prices from FRED (only last 10 years of daily data is freely available)<jupyter_code>df = web.DataReader(name='SP500', data_source='fred', start=2009).squeeze().to_frame('close')
print(df.info())
with pd.HDFStore(DATA_STORE) as store:
    store.put('sp500/fred', df)
sp500_stooq = (pd.read_csv('^spx_d.csv', index_col=0,
                     parse_dates=True).loc['1950':'2019'].rename(columns=str.lower))
print(sp500_stooq.info())
df = pd.read_csv('us_equities_meta_data.csv')
df.info()

with pd.HDFStore(DATA_STORE) as store:
    store.put('us_equities/stocks', df.set_index('ticker'))<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 6834 entries, 0 to 6833
Data columns (total 7 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   ticker     6834 non-null   object 
 1   name       6834 non-null   object 
 2   lastsale   6718 non-null   float64
 3   marketcap  5766 non-null   float64
 4   ipoyear    3038 non-null   float64
 5   sector     5288 non-null   object 
 6   industry   5288 non-null   object 
dtypes: float64(3), object(4)
memory usage: 373.9+ KB
{'fraction_non_alphanumeric': 0.10385756676557864, 'fraction_numerical': 0.030586624058434146, 'mean_word_length': 4.222884386174017, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '496249', 'license': 'MIT', 'n_tokens_mistral': 1476, 'n_tokens_neox': 1355, 'n_words': 441, 'path': 'data/my_create_datasets.ipynb', 'repo_name': 'PerrySong/ML4T', 'size': '4381'}