<jupyter_start><jupyter_text>Stock Market Prediction Experiments<NAME>, PhD Goal: Predict whether future daily returns are going to be positive or negative.This is a binary classification.$Return_i=\dfrac{AdjClose_{i}–AdjClose_{i−1}}{AdjClose_{i−1}}$<jupyter_code>import numpy as np
import pandas as pd
import datetime
from sklearn import preprocessing
from datetime import datetime
from sklearn.ensemble import RandomForestClassifier
from sklearn import neighbors
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.svm import SVC
import operator
import pandas.io.data
#import pandas_datareader.data # need to update
from sklearn.qda import QDA
import re
from dateutil import parser
from backtest import Strategy, Portfolio<jupyter_output><empty_output><jupyter_text>Predictorsthe following predictors have been selected:- [NASDAQ Composite (^IXIC Yahoo Finance)](http://finance.yahoo.com/q?s=^IXIC&ql=1)  Dow Jones Industrial Average (^DJI Quandl)   Frankfurt DAX (^GDAXI Yahoo Finance)  London FTSE-100 (^FTSE Yahoo Finance)  Paris CAC 40 (^FCHI Yahoo Finance)  Tokyo Nikkei-225 (^N225 Yahoo Finance)  Hong Kong Hang Seng (^HSI Yahoo Finance)  Australia ASX-200 (^AXJO Yahoo Finance)   So to recap the logic is the following:1) Download 9 dataframes (NASDAQ, Dow Jones, Frankfurt, London , Paris, Tokyo, Hong Kong, Australia, S&P 500).  2) Compute S&P 500 daily returns and turn them into a binary variable (Up, Down). This is my output and it won’t be touched anymore.  3) Play with all the other columns of the 9 available dataframes (S&P 500 included) as explained in the following post.<jupyter_code>def getStock(symbol, start, end):
    """
    Downloads Stock from Yahoo Finance.
    Computes daily Returns based on Adj Close.
    Returns pandas dataframe.
    """
    df =  pd.io.data.get_data_yahoo(symbol, start, end)

    df.columns.values[-1] = 'AdjClose'
    df.columns = df.columns + '_' + symbol
    df['Return_%s' %symbol] = df['AdjClose_%s' %symbol].pct_change()
    
    return df
def getStockFromQuandl(symbol, name, start, end):
    """
    Downloads Stock from Quandl.
    Computes daily Returns based on Adj Close.
    Returns pandas dataframe.
    """
    import Quandl
    df =  Quandl.get(symbol, trim_start = start, trim_end = end, authtoken="your token")
 
    df.columns.values[-1] = 'AdjClose'
    df.columns = df.columns + '_' + name
    df['Return_%s' %name] = df['AdjClose_%s' %name].pct_change()
    
    return df
def getStockDataFromWeb(fout, start_string, end_string):
    """
    Collects predictors data from Yahoo Finance and Quandl.
    Returns a list of dataframes.
    """
    start = parser.parse(start_string)
    end = parser.parse(end_string)
    
    nasdaq = getStock('^IXIC', start, end)
    frankfurt = getStock('^GDAXI', start, end)
    london = getStock('^FTSE', start, end)
    paris = getStock('^FCHI', start, end)
    hkong = getStock('^HSI', start, end)
    nikkei = getStock('^N225', start, end)
    australia = getStock('^AXJO', start, end)
    
    djia = getStockFromQuandl("YAHOO/INDEX_DJI", 'Djia', start_string, end_string) 
    
    out =  pd.io.data.get_data_yahoo(fout, start, end)
    out.columns.values[-1] = 'AdjClose'
    out.columns = out.columns + '_Out'
    out['Return_Out'] = out['AdjClose_Out'].pct_change()
    
    return [out, nasdaq, djia, frankfurt, london, paris, hkong, nikkei, australia]
import pandas as pd
from pandas.io.data import DataReader
from datetime import datetime

goog = DataReader("GOOG",  "yahoo", datetime(2000,1,1), datetime(2012,1,1))
print goog[:100]["Adj Close"]

symbols_list = ['ORCL', 'TSLA', 'IBM','YELP', 'MSFT']
d = {}
for ticker in symbols_list:
    d[ticker] = DataReader(ticker, "yahoo", '2014-12-01')
    print type(d['ORCL'])
    print( d['ORCL'] )
pan = pd.Panel(d)
df1 = pan.minor_xs('Adj Close')
print(df1)<jupyter_output>Date
2004-08-19     50.119968
2004-08-20     54.100990
2004-08-23     54.645447
2004-08-24     52.382705
2004-08-25     52.947145
2004-08-26     53.901190
2004-08-27     53.022069
2004-08-30     50.954132
2004-08-31     51.133953
2004-09-01     50.075011
2004-09-02     50.704382
2004-09-03     49.955130
2004-09-07     50.739348
2004-09-08     51.098988
2004-09-09     51.103983
2004-09-10     52.612476
2004-09-13     53.696398
2004-09-14     55.689407
2004-09-15     55.944152
2004-09-16     56.928171
2004-09-17     58.686414
2004-09-20     59.620484
2004-09-21     58.861240
2004-09-22     59.130972
2004-09-23     60.349754
2004-09-24     59.855246
2004-09-27     59.071031
2004-09-28     63.366744
2004-09-29     65.474638
2004-09-30     64.735377
                 ...    
2004-11-29     90.434722
2004-11-30     90.899259
2004-12-01     89.890261
2004-12-02     89.610541
2004-12-03     90.110048
2004-12-06     88.057091
2004-12-07     85.629519
2004-12-08     84.905237
2004-12-09     86.62[...]<jupyter_text>Feature Selection- AdjClose_sp  - AdjClose_nasdaq  - AdjClose_djia  - AdjClose_frankfurt  - AdjClose_london  - AdjClose_paris  - AdjClose_nikkei  - AdjClose_hkong  - AdjClose_australia  To account for time variations, add the following basic financial metrics:**1) Days Returns:** percentage difference of Adjusted Close Price of $i^{th}$ day  compared to $(i-1)^{th}$ day. $Return_i=\dfrac{AdjClose_i–AdjClose_{i−1}}{AdjClose_{i−1}}$**2) Multiple Day Returns:** percentage difference of the *Adjusted Close Price* of $i^{th}$ day  compared to $(i-\delta)^{th}$ day. Example: 3-days Return is the percentage difference of Adjusted Close Price of today compared to the one of 3 days ago.$Return_{\delta}=\dfrac{AdjClose_i–AdjClose_{i−\delta}}{AdjClose_{i−\delta}}$**3) Moving Average Returns:** average returns over last $\delta$ days. Example: 3-day return is the percentage difference of the *Adjusted Close Price* of today compared to the one of 3 days ago.$MovingAverage_{\delta}=\dfrac{Return_i+Return_{i−1}+Return_{i−2}+…+Return_{i−\delta}}{\delta}$**4) Time Lagged Returns:** shift the daily returns *n* days backwards. Example: if n =  1 todays’ Return becomes yesterdays’ Return.**Summary:**- Start with 8 basic predictors: (the Adjusted Close Price of the 8 world major stock indices) + 1 output/predictor (Adjusted Close Price of S&P 500). - The daily returns of each predictor.- Add features to the DataFrame using the 4 financial metrics. With values for $n$ and $\delta$ to generate additional features.- Remove the original Adjusted Close Price, ending up with a perfectly scaled dataset. - Notice that a bunch of missing values are automatically produced. To make this point clear let’s walk through a practical example:What happens you compute the 3-day-Moving Average on the Daily Returns column? Pandas is going to replace today’s return with the average of the returns of the last 3 days. Now let’s suppose that the first entry in the dataframe corresponds to 20 April 2014. There is going to be no 3-day-Moving Average for 20 April 2014 as there are no 3 previous days. The same for 21 and 22 April 2014. Actually the first non-missing day is going to be 23 April 2014 as it would be possible to compute the average of daily returns on the 3 previous days. Notice that the same issue (with slightly different results) rises with the other financial metrics are taken into account.Given the AdjClose and Returns, addFeatures() returns delta-Multiple Day Returns and delta-Returns Moving Average. This function is called for several deltas inside applyRollMeanDelayedReturns()<jupyter_code>def addFeatures(dataframe, adjclose, returns, n):
    """
    operates on two columns of dataframe:
    - n >= 2
    - given Return_* computes the return of day i respect to day i-n. 
    - given AdjClose_* computes its moving average on n days

    """
    
    return_n = adjclose[9:] + "Time" + str(n)
    dataframe[return_n] = dataframe[adjclose].pct_change(n)
    
    roll_n = returns[7:] + "RolMean" + str(n)
    dataframe[roll_n] = pd.rolling_mean(dataframe[returns], n)<jupyter_output><empty_output><jupyter_text>Given the list of datasets and the range of deltas to explore, applyRollMeanDelayedReturns() adds features to each dataset and returns the augmented list.<jupyter_code>def applyRollMeanDelayedReturns(datasets, delta):
    """
    applies rolling mean and delayed returns to each dataframe in the list
    """
    for dataset in datasets:
        columns = dataset.columns    
        adjclose = columns[-2]
        returns = columns[-1]
        for n in delta:
            addFeatures(dataset, adjclose, returns, n)
    
    return datasets<jupyter_output><empty_output><jupyter_text>– mergeDataframes() is fundamental as it takes the list of augmented datasets produced by applyRollMeanDelayedReturns() and merges all of them in the finance dataframe applying a time cut at the beginning of the series (all data after 1993 – I decided to implement this time cut as Australia ASX-200 data is not available before that time) and selecting only the relevant columns out of each dataframe. This is the step in which we get rid of all the Open, High, Low, Close, Volume, AdjClose columns in each dataset and keep only the Daily Returns, delta-Multiple Day Returns and delta-Returns Moving Average previously created.I want to stress the specific merging Pandas command. This step is quite tricky as I’m concatenating dataframes by date index. The issue arising is that markets all over the world have not the same trading days due basically to different national holidays. So there is going to be an unavoidable annoying mismatch between dates. I faced the issue in the following way:Perform an OUTER JOIN of all the predictors. This command generates a UNION of all the columns, thus creating a bunch of NAs. I afterwards imputed them by linear interpolation. Let’s call the result of this operation PREDICTORS.Perform a S&P LEFT JOIN PREDICTORS. This line shrinks PREDICTORS to match S&P date index. Thus no additional NA will be created in the output dataframe.<jupyter_code>def mergeDataframes(datasets, index, cut):
    """
    merges datasets in the list 
    """
    subset = []tion
    subset = [dataset.iloc[:, index:] for dataset in datasets[1:]]
    
    first = subset[0].join(subset[1:], how = 'outer')
    finance = datasets[0].iloc[:, index:].join(first, how = 'left') 
    finance = finance[finance.index > cut]
    return finance
def applyTimeLag(dataset, lags, delta):
    """
    apply time lag to return columns selected according  to delta.
    Days to lag are contained in the lads list passed as argument.
    Returns a NaN free dataset obtained cutting the lagged dataset
    at head and tail
    """
    
    dataset.Return_Out = dataset.Return_Out.shift(-1)
    maxLag = max(lags)

    columns = dataset.columns[::(2*max(delta)-1)]
    for column in columns:
        for lag in lags:
            newcolumn = column + str(lag)
            dataset[newcolumn] = dataset[column].shift(lag)

    return dataset.iloc[maxLag:-1,:]<jupyter_output><empty_output><jupyter_text>Prepare data for classification<jupyter_code>def applyTimeLag(dataset, lags, delta):
    """
    apply time lag to return columns selected according  to delta.
    Days to lag are contained in the lads list passed as argument.
    Returns a NaN free dataset obtained cutting the lagged dataset
    at head and tail
    """
    
    dataset.Return_Out = dataset.Return_Out.shift(-1)
    maxLag = max(lags)

    columns = dataset.columns[::(2*max(delta)-1)]
    for column in columns:
        for lag in lags:
            newcolumn = column + str(lag)
            dataset[newcolumn] = dataset[column].shift(lag)

    return dataset.iloc[maxLag:-1,:]<jupyter_output><empty_output><jupyter_text>performClassification()<jupyter_code>def performClassification(X_train, y_train, X_test, y_test, method, parameters, fout, savemodel):
    """
    performs classification on daily returns using several algorithms (method).
    method --> string algorithm
    parameters --> list of parameters passed to the classifier (if any)
    fout --> string with name of stock to be predicted
    savemodel --> boolean. If TRUE saves the model to pickle file
    """
   
    if method == 'RF':   
        return performRFClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel)
        
    elif method == 'KNN':
        return performKNNClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel)
    
    elif method == 'SVM':   
        return performSVMClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel)
    
    elif method == 'ADA':
        return performAdaBoostClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel)
    
    elif method == 'GTB': 
        return performGTBClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel)

    elif method == 'QDA': 
        return performQDAClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel)<jupyter_output><empty_output><jupyter_text>Random Forest<jupyter_code>def performRFClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel):
    """
    Random Forest Binary Classification
    """
    clf = RandomForestClassifier(n_estimators=1000, n_jobs=-1)
    clf.fit(X_train, y_train)
    
    if savemodel == True:
        fname_out = '{}-{}.pickle'.format(fout, datetime.now())
        with open(fname_out, 'wb') as f:
            cPickle.dump(clf, f, -1)    
    
    accuracy = clf.score(X_test, y_test)
    
    return accuracy<jupyter_output><empty_output><jupyter_text>K-Nearest Neighbors<jupyter_code>def performKNNClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel):
    """
    KNN binary Classification
    """
    clf = neighbors.KNeighborsClassifier()
    clf.fit(X_train, y_train)

    if savemodel == True:
        fname_out = '{}-{}.pickle'.format(fout, datetime.now())
        with open(fname_out, 'wb') as f:
            cPickle.dump(clf, f, -1)    
    
    accuracy = clf.score(X_test, y_test)
    
    return accuracy<jupyter_output><empty_output><jupyter_text>SVM<jupyter_code>def performSVMClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel):
    """
    SVM binary Classification
    """
    c = parameters[0]
    g =  parameters[1]
    clf = SVC()
    clf.fit(X_train, y_train)

    if savemodel == True:
        fname_out = '{}-{}.pickle'.format(fout, datetime.now())
        with open(fname_out, 'wb') as f:
            cPickle.dump(clf, f, -1)    
    
    accuracy = clf.score(X_test, y_test)
    
    return accuracy<jupyter_output><empty_output><jupyter_text>Adaptive Boosting<jupyter_code>def performAdaBoostClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel):
    """
    Ada Boosting binary Classification
    """
    n = parameters[0]
    l =  parameters[1]
    clf = AdaBoostClassifier()
    clf.fit(X_train, y_train)

    if savemodel == True:
        fname_out = '{}-{}.pickle'.format(fout, datetime.now())
        with open(fname_out, 'wb') as f:
            cPickle.dump(clf, f, -1)    
    
    accuracy = clf.score(X_test, y_test)
    
    return accuracy<jupyter_output><empty_output><jupyter_text>Gradient Boosting<jupyter_code>def performGTBClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel):
    """
    Gradient Tree Boosting binary Classification
    """
    clf = GradientBoostingClassifier(n_estimators=100)
    clf.fit(X_train, y_train)

    if savemodel == True:
        fname_out = '{}-{}.pickle'.format(fout, datetime.now())
        with open(fname_out, 'wb') as f:
            cPickle.dump(clf, f, -1)    
    
    accuracy = clf.score(X_test, y_test)
    
    return accuracy<jupyter_output><empty_output><jupyter_text>Quadratic Discriminant Analysis<jupyter_code>def performQDAClass(X_train, y_train, X_test, y_test, parameters, fout, savemodel):
    """
    Quadratic Discriminant Analysis binary Classification
    """
    def replaceTiny(x):
        if (abs(x) < 0.0001):
            x = 0.0001
    
    X_train = X_train.apply(replaceTiny)
    X_test = X_test.apply(replaceTiny)
    
    clf = QDA()
    clf.fit(X_train, y_train)

    if savemodel == True:
        fname_out = '{}-{}.pickle'.format(fout, datetime.now())
        with open(fname_out, 'wb') as f:
            cPickle.dump(clf, f, -1)    
    
    accuracy = clf.score(X_test, y_test)
    
    return accuracy<jupyter_output><empty_output><jupyter_text>Perform feature selection<jupyter_code>def performFeatureSelection(maxdeltas, maxlags, fout, cut, start_test, path_datasets, savemodel, method, folds, parameters):
    """
    Performs Feature selection for a specific algorithm
    """
    
    for maxlag in range(3, maxlags + 2):
        lags = range(2, maxlag) 
        print ''
        print '============================================================='
        print 'Maximum time lag applied', max(lags)
        print ''
        for maxdelta in range(3, maxdeltas + 2):
            datasets = loadDatasets(path_datasets, fout)
            delta = range(2, maxdelta) 
            print 'Delta days accounted: ', max(delta)
            datasets = applyRollMeanDelayedReturns(datasets, delta)
            finance = mergeDataframes(datasets, 6, cut)
            print 'Size of data frame: ', finance.shape
            print 'Number of NaN after merging: ', count_missing(finance)
            finance = finance.interpolate(method='linear')
            print 'Number of NaN after time interpolation: ', count_missing(finance)
            finance = finance.fillna(finance.mean())
            print 'Number of NaN after mean interpolation: ', count_missing(finance)    
            finance = applyTimeLag(finance, lags, delta)
            print 'Number of NaN after temporal shifting: ', count_missing(finance)
            print 'Size of data frame after feature creation: ', finance.shape
            X_train, y_train, X_test, y_test  = prepareDataForClassification(finance, start_test)
            
            print performCV(X_train, y_train, folds, method, parameters, fout, savemodel)
            print ''<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10328638497652583, 'fraction_numerical': 0.04380005523336095, 'mean_word_length': 3.4950347567030784, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '475800', 'license': 'MIT', 'n_tokens_mistral': 6322, 'n_tokens_neox': 5689, 'n_words': 1780, 'path': 'labs/lab10_time_series/Stock Prediction Experiments.ipynb', 'repo_name': 'larsonma/DataScienceIntro', 'size': '18109'}