<jupyter_start><jupyter_text>Intraday Trading via Day Trading Techniques & Indicators--- Data collected via AlphaVantage free API using extended intraday data. > https://www.alphavantage.co/documentation/---  00 - Data Collection  Library Imports<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read in sample Data**First, we want to ensure our data is in the proper form and there is no extra cleaning that needs to be done. For this we will look at our most recent dataset.**<jupyter_code>df = pd.read_csv('../01_Data/extended_intraday_SPY_1min_year1month1_adjusted.csv')
df.head()<jupyter_output><empty_output><jupyter_text>**We want our index to be the 'time' columns and make sure it is in the form of a DateTime object.**<jupyter_code>df.set_index(pd.DatetimeIndex(df['time']), inplace=True)
df.drop(columns = ['time'], inplace = True)
df.head()<jupyter_output><empty_output><jupyter_text>**We shouldn't have any null values given how the data was acquired, but we should always check.**<jupyter_code>df.info()
df.describe()<jupyter_output><empty_output><jupyter_text>**Finally, our data should be for the most recent month period, from 9/17 to 10/17 as of the date it was acquired. Furthermore it should include off-hours data.**<jupyter_code>df[['close']].plot(figsize=(18,9));<jupyter_output><empty_output><jupyter_text>**Everything seems in order. These 'straight lines' extending from untracked overnight and weekends will likely need to be removed in order to preserve the integrity of our derived columns in the future; ex. VWAP and other Moving Averages.**  Read in all Datasets**We have 2 years worth of data in the form of year1month1 referring to 1 month prior and year2month12 referring to 2 years prior, with each dataset containing 1 month of prices and volume for SPY.**<jupyter_code># Year we want
y = 1
# Month we want; we already imported month 1 above
m = 2

while (y < 3) and (m < 13):
    temp = pd.read_csv('../01_Data/extended_intraday_SPY_1min_year' + str(y) + 'month' + str(m) + '_adjusted.csv')
    temp.set_index(pd.DatetimeIndex(temp['time']), inplace=True)
    temp.drop(columns = ['time'], inplace = True)
    
    df = pd.concat([df, temp])
    
    if (y == 1) and (m == 12):
        y = 2
        m = 0
    
    m += 1

df.shape<jupyter_output><empty_output><jupyter_text>**Shape appears to be correct given size of individual datasets.****To be sure, let's plot the data.**<jupyter_code>df[['close']].plot(figsize=(18,9), title='SPY 1minute on 2 Year Period');<jupyter_output><empty_output><jupyter_text>**When compared to a graph found on TradingView, or any online resource, we see that this graph does represent the SPDR S&P500 ETF TRUST (SPY)**> Therefore we know our data was imported correctly---  Sort Index**Currently our dataframe starts at our most recent date and moves backwards. This will cause issue with moving averages we want to make so we need to sort our index so the earliest dates are first.**<jupyter_code>df = df.sort_index()
df.head(25)<jupyter_output><empty_output><jupyter_text>**Let's make sure there were no unintended consequences**<jupyter_code>df[['close']].plot(figsize=(18,9), title='SPY 1minute on 2 Year Period');<jupyter_output><empty_output><jupyter_text>**Looks good.**  Save Combined Dataset<jupyter_code>df.to_csv('../01_Data/extended_intraday_SPY_1min_combined.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10866456963111239, 'fraction_numerical': 0.01687160423219903, 'mean_word_length': 5.477777777777778, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '209747', 'license': 'MIT', 'n_tokens_mistral': 1169, 'n_tokens_neox': 1105, 'n_words': 382, 'path': '00_Code/00-Data_Collection.ipynb', 'repo_name': 'bvarnam/StockPrediction', 'size': '3497'}