<jupyter_start><jupyter_text>Bitcoin Battle Royale - Getting started (Python 2 notebook) Welcome to Bitcoin Battle Royale, a contest where you have the predict the probability of bitcoin moving more than 0.25% over the next 30min.This notebook aims to provide a few code snippets you can use to download data from s3, and also a few ways to interact with the feature set.You don't have to use any of this code, this should only be considered as a quick start guide...If you have any questions, ask Eben or Mike :)  How to download data from S3 This notebook references a config.ini file in the config folder. You should enter your AWS access key and secret key in the config.ini file. If you wanted to use ConfigParser to load your credentials, this is what it would look like:<jupyter_code>%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import statsmodels.graphics.tsaplots as tsa
import configparser, os

Config = configparser.ConfigParser(allow_no_value=True)
Config.read(os.path.join(os.path.dirname(os.getcwd()), 'config', 'config.ini'))

aws_access_key_id = Config.get("aws_credentials", "aws_access_key_id")
aws_secret_access_key = Config.get("aws_credentials", "aws_secret_access_key")
bucket_name_value = Config.get("aws_credentials", "bucket_name_value")<jupyter_output><empty_output><jupyter_text>To download data, we're going to create a download folder on our local machine...<jupyter_code>download_folder = os.path.join(os.path.dirname(os.getcwd()), 'data')
if not os.path.exists(download_folder):
    os.makedirs(download_folder)
    print("Created download folder", download_folder)
else:
    print("Download folder already exists", download_folder)<jupyter_output>Download folder already exists C:\Users\<NAME>\Documents\Projects\btc_battle\data<jupyter_text>Great! Now we're ready to download data from s3.The following function makes it easy to list all the bitcoin files available for download in our s3 bucket...<jupyter_code>import boto
from boto.s3.key import Key
from boto.s3.connection import OrdinaryCallingFormat

def download_most_recent_tick_data_files(download_destination=download_folder,
                                         number_of_files_to_download=5):
    
    # establish a connection using our access keys...
    conn = boto.connect_s3(aws_access_key_id,
                           aws_secret_access_key,
                           calling_format=OrdinaryCallingFormat())
    # connect to our bucket...
    bucket = conn.get_bucket(bucket_name_value)

    # connect to the folder that contains all the bitcoin trading data...
    most_recent_files = sorted([k.key for k in boto.s3.bucketlistresultset.bucket_lister(bucket,
                                                                                         prefix="featuredata/GDAX/")],
                               reverse=True)[0:int(number_of_files_to_download)]
    for most_recent_file_key in most_recent_files:
        k = Key(bucket, most_recent_file_key)
        k.get_contents_to_filename(os.path.join(download_destination,
                                                most_recent_file_key.split("/")[-1]))
        print("Downloaded to local...", most_recent_file_key)

# call the function...
download_most_recent_tick_data_files()<jupyter_output>('Downloaded to local...', u'featuredata/GDAX/2018-04-25-GDAX.finalfeatures.csv')
('Downloaded to local...', u'featuredata/GDAX/2018-04-24-GDAX.finalfeatures.csv')
('Downloaded to local...', u'featuredata/GDAX/2018-04-23-GDAX.finalfeatures.csv')
('Downloaded to local...', u'featuredata/GDAX/2018-04-22-GDAX.finalfeatures.csv')
('Downloaded to local...', u'featuredata/GDAX/2018-04-21-GDAX.finalfeatures.csv')<jupyter_text>Quick description of features...<jupyter_code>import pandas as pd

# pick the most recent file for analysis...
most_recent_tick_data_file = os.path.join(download_folder,
                                          sorted(os.listdir(download_folder), reverse=True)[0])
data_df = pd.read_csv(most_recent_tick_data_file)
print(most_recent_tick_data_file, "df loaded, shape", data_df.shape)<jupyter_output>C:\Users\<NAME>\Documents\Projects\btc_battle\data\2017-12-16-GDAX.finalfeatures.csv df loaded, shape (60761, 115)<jupyter_text>Plot of prices<jupyter_code># convert times to datetimes
data_df['time'] = pd.to_datetime(data_df['time'])
from datetime import timedelta

def hhmax(row):         
    ser = data_df['price'][(data_df['time'] > row) &
               (data_df['time'] <= row + timedelta(minutes=30))]
    return ser.max()

data_df['MaxY'] = data_df['time'].apply(hhmax)

from datetime import timedelta
import numpy as np

def hhmax(row):         
    ser = data_df['price'][(data_df['time'] > row) &
               (data_df['time'] <= row + timedelta(minutes=30))]
    
    minY = np.min(ser)
    maxY = np.max(ser)
    currentY = ser[0]
    
    thresh_low = currentY * (1 - 0.0025)
    thres_hi = currentY * (1 + 0.0025)
    
    exceeds_low == minY <= thresh_low
    exceeds_hi == maxY >= thresh_hi
    
    if (not exceeds_low) and (not exceeds_hi):
        target = 0
    elif (exceeds_low) and (not exceeds_hi):
        target = 1
    elif (not exceeds_low) and (exceeds_hi):
        target = 2
    else: 
        target = 3
                    
    return target

data_df['MinY'] = data_df['time'].apply(hhmax)

if 1==0:
    a = 1
elif 1 == 1:
    a = 2
# define plotting function

def tsplot2(y, title, lags=None, figsize=(12, 8)):
    """
    Credit to <NAME>, PyData NYC 2017, "Time Series Forecasting using Statistical and Machine Learning Models"
    Examine the patterns of  ACF and PACF, along with the time series plot and histogram.
    :param y:
    :param title:
    :param lags:
    :param figsize:
    :return:
    """

    fig = plt.figure(figsize=figsize)
    layout = (2,2)
    ts_ax = plt.subplot2grid(layout,  (0,0))
    hist_ax = plt.subplot2grid(layout, (0, 1))
    acf_ax = plt.subplot2grid(layout, (1, 0))
    pacf_ax = plt.subplot2grid(layout, (1, 1))

    y.plot(ax=ts_ax)
    ts_ax.set_title(title, fontsize=14, fontweight='bold')
    y.plot(ax=hist_ax, kind='hist', bins=25)
    hist_ax.set_title('Histogram')
    tsa.plot_acf(y, lags=lags, ax=acf_ax)
    tsa.plot_pacf(y, lags=lags, ax=pacf_ax)
    [ax.set_xlim(0) for ax in [acf_ax, pacf_ax]]
    sns.despine()
    plt.tight_layout()
    return ts_ax, acf_ax, pacf_ax
data = pd.read_csv('target_test.csv')
layout = (1,2)
price_ax = plt.subplot2grid(layout,  (0,0))
target_ax = plt.subplot2grid(layout,  (0,1))


data['price'][:10000].plot(ax=price_ax)
data['target'][:10000].plot(ax=target_ax)
# for each series, plot a time series, histogram, autocorr, and partial autocorr
tsplot2(data_df['target'].dropna(), title='MaxY', lags=100)
# we could just dump all the feature names in alphabetical order...
for feature in sorted(data_df.columns.values):
    print(feature)<jupyter_output>0th_larg_order
0th_larg_order_1000_rar
0th_larg_order_100_rar
0th_larg_order_10_rar
1th_larg_order
1th_larg_order_1000_rar
1th_larg_order_100_rar
1th_larg_order_10_rar
2th_larg_order
2th_larg_order_1000_rar
2th_larg_order_100_rar
2th_larg_order_10_rar
3th_larg_order
3th_larg_order_1000_rar
3th_larg_order_100_rar
3th_larg_order_10_rar
4th_larg_order
4th_larg_order_1000_rar
4th_larg_order_100_rar
4th_larg_order_10_rar
5th_larg_order
5th_larg_order_1000_rar
5th_larg_order_100_rar
5th_larg_order_10_rar
6th_larg_order
6th_larg_order_1000_rar
6th_larg_order_100_rar
6th_larg_order_10_rar
7th_larg_order
7th_larg_order_1000_rar
7th_larg_order_100_rar
7th_larg_order_10_rar
8th_larg_order
8th_larg_order_1000_rar
8th_larg_order_100_rar
8th_larg_order_10_rar
9th_larg_order
9th_larg_order_1000_rar
9th_larg_order_100_rar
9th_larg_order_10_rar
Unnamed: 0
last_size
last_size_1000_rar
last_size_100_rar
last_size_10_rar
max_price
max_price_1000_rar
max_price_100_rar
max_price_10_rar
max_volume
max_volume[...]<jupyter_text>Key points:1. Whenever you see a _rar value, remember that it refers to rolling average return by time, e.g. 10m = 10 minutes, 1m = 1 minute, 60m = an hour...2. "larg_order" features refer to the orderbook on GDAX, largest orders, already normalized3. "last_size" refers to the size of the most recent trade in BTC4. "max_price", "min_price", "std_price", "max_volume", "mean_price" etc. all refers to statistical properties of recent price and volume trends5. "side" is a variable that shows if the recent trade was a buy order or a sell order...6. "time" is the UTC timestamp of the trade7. "volume_over_price" and "volume_under_price" - ?8. "vwp" refers to volume weighted price  Display plots for feature interaction<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = [10, 10]

feature_categories_to_plot = ["_larg_order_10_rar", "_price_10_rar", "_volume_10_rar"]

for feature_category in feature_categories_to_plot:
    features_to_scan = sorted([h for h in data_df.columns.values if h.find(feature_category) != -1])[0:4]
    data_df[features_to_scan].hist()<jupyter_output><empty_output><jupyter_text>...notice how very few of these features look like they have a "normal" distribution...this is likely due to the impact of outliers...One quick and easy way to filter for outliers is to simply remove the bottom x% and the top x% of a feature set...<jupyter_code>import numpy as np
import warnings

warnings.filterwarnings('ignore')

feature_categories_to_plot = ["_larg_order_10_rar", "_price_10_rar", "_volume_10_rar"]

for feature_category in feature_categories_to_plot:
    features_to_scan = sorted([h for h in data_df.columns.values if h.find(feature_category) != -1])[0:4]
    outliers_removed_df = data_df[features_to_scan].copy()
    outliers_removed_df.dropna(inplace=True)
    for feature_to_scan in features_to_scan:
        lower_bound = np.percentile(outliers_removed_df[feature_to_scan], 10)
        upper_bound = np.percentile(outliers_removed_df[feature_to_scan], 90)
        print(feature_to_scan, "lower_bound", lower_bound, "upper_bound", upper_bound)
        outliers_removed_df[feature_to_scan] = outliers_removed_df[feature_to_scan].map(lambda x: lower_bound if x < lower_bound
                                                                                        else upper_bound if x > upper_bound
                                                                                        else x)
    outliers_removed_df[features_to_scan].hist()
# some of the feature sets have very high correlations with other feature sets...
# this is a function that takes a feature set as a string tag and generates correlation matrices

import seaborn as sns

def generate_correlation_plot(feature_category_to_plot):
    correlation_df = data_df.copy()
    headers_to_plot = sorted([h for h in correlation_df.columns.values if h.find(feature_category_to_plot) != -1])
    correlation_df = correlation_df[headers_to_plot]
    correlation_df.dropna(inplace=True)
    corr = correlation_df.corr()
    sns.heatmap(corr,
                xticklabels=corr.columns.values,
                yticklabels=corr.columns.values)
generate_correlation_plot("_volume_")
generate_correlation_plot("_price_")
generate_correlation_plot("_larg_order_10_")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09409151488227455, 'fraction_numerical': 0.03056419369169258, 'mean_word_length': 3.9982238010657194, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 47, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '494043', 'license': 'MIT', 'n_tokens_mistral': 3949, 'n_tokens_neox': 3705, 'n_words': 916, 'path': 'notebooks/feature_analysis.ipynb', 'repo_name': 'braddeutsch/example_pipeline', 'size': '11272'}