<jupyter_start><jupyter_text>A classification project for time-series data using Random Forest classifier<jupyter_code>#import libraries
import yfinance as yf
import pandas as pd
import talib
from talib import abstract #conda install -c conda-forge ta-lib
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import MinMaxScaler
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier as rfc
from sklearn.feature_selection import RFE, RFECV
from sklearn.metrics import confusion_matrix, roc_curve, auc, classification_report
import joblib

pd.set_option('display.max_rows', 600)
#import yahoo stock data
data = yf.Ticker('SPY').history(start='1990-01-29') #get the yahoo stock data start from the beginning
data.columns = ['open','high','low','close','volume','dividends','split'] #renamce columns
data.head()
#data pre-processing of cyclical features
#this step helps classification on time-series data
#source: http://blog.davidkaleko.com/feature-engineering-cyclical-features.html
#note year is not cyclical features
data['year'] = data.index.to_series().dt.year.to_list()
data['month'] = data.index.to_series().dt.month.to_list()
data['day'] = data.index.to_series().dt.day.to_list()
data['weekday'] = data.index.to_series().dt.dayofweek.to_list()

data['month_sin'] = np.sin(data.month*(2.*np.pi/12))
data['month_cos'] = np.cos(data.month*(2.*np.pi/12))

data['day_sin'] = np.sin(data.day*(2.*np.pi/30))
data['day_cos'] = np.cos(data.day*(2.*np.pi/30))

data['weekday_sin'] = np.sin(data.weekday*(2.*np.pi/7))
data['weekday_cos'] = np.cos(data.weekday*(2.*np.pi/7))

data.head()<jupyter_output><empty_output><jupyter_text>The date information is extracted from the index column and its relative trigonometry functions are also computed for time-series classification.<jupyter_code>#compute indicators for the dataset
ind_list = talib.get_functions() #get all indicators for financial analysis

temp = data
#compute each indicator for the dataset
#source: 
for ind in ind_list:
    if ind != 'MAVP': #MAVP needs periods
        output = eval('abstract.' + ind + '(data)')
    else:
        output = eval('talib.MAVP(data.close, periods=data.index.to_series(), minperiod=2, maxperiod=30, matype=0)')
    output.name = ind.lower() if type(output) == pd.core.series.Series else None #rename for 1D output
    temp = pd.merge(temp, pd.DataFrame(output), 
                    left_on = temp.index, 
                    right_on = output.index).set_index('key_0') #merge output with original data
ind_data = temp
ind_data.head()<jupyter_output><empty_output><jupyter_text>All the indices from talib are computed. This increases the degree of freedom when performing feature selection as more combinations of features are provided. However, more or less features does not necessarily mean better or worse prediction.<jupyter_code>#check null
ind_data.isnull().sum().describe()<jupyter_output><empty_output><jupyter_text>The statistics tells that may indices contains null/na/nan values<jupyter_code>#remove column that has a lot of null (3% of the sample number)
#otherwise replace null by mean
clean_data = ind_data
for col in clean_data.columns:
    if clean_data[col].isnull().sum() > 0.03*len(clean_data):
        clean_data.drop(col, axis=1, inplace=True)
clean_data = clean_data.fillna(clean_data.mean())
clean_data.head()<jupyter_output><empty_output><jupyter_text>As the nulls occurs within the indices not the original data, replacing some of them rather than removing all of them maintains the variability of the dataset. Other than mean, median or mod can also be used to replace the null.<jupyter_code>#visua;ize the dataset
clean_data.plot(style='.', subplots=True, legend=False, use_index=False, alpha=0.3,
                title=list(clean_data.columns), layout=(19,10), figsize=(15,32)); #colormap="winter"<jupyter_output><empty_output><jupyter_text>The visualizations show that there are certain 'outliers' lie within some features such as 'max_x' and 'max_y'. However, outliers do not necessarily mean they are invalid data points. Moreover, there are some features exhibiting triviality as they contain only same values. Performing feature selection can remove the unimportant features. The appropriate algorithm will be using tree algorithms such as Random Forest which are not sensitive to outliers.<jupyter_code>#scale the data between -1 and 1
#useful for svm or knn
#little effect on random forest
transformed_data = clean_data
scaler = MinMaxScaler(feature_range=(-1,1))
transformed_data.iloc[:,:] = scaler.fit_transform(transformed_data)
transformed_data.head()<jupyter_output><empty_output><jupyter_text>All the features are standardize that they range between -1 and 1. This step is trivial for Random Forest or Decision Tree classifier, while it helps improve the performance when using SVM or KNN classifier which uses distance vector.<jupyter_code>#create new column to indicate rise/drop after 5 days
#observed that rise is about 1000 more than drop
transformed_data['target'] = np.where(transformed_data.close.shift(-5) > transformed_data.close, 1, -1)
transformed_data['target'].value_counts()<jupyter_output><empty_output><jupyter_text>It can seen that the difference in number is more than 1000. This means that there will be some bias in the dataset, which will affect the classification.<jupyter_code>#pearson correlation
cor = transformed_data.corr()
abs(cor['target']).sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>The pearson correlation is used to investigate the direct correlation between each independent variable and traget variable. The features that give NaN in pearson coefficient will be removed.<jupyter_code>#remove uncorrelated features
transformed_data = transformed_data[cor['target'].dropna().index.to_list()]
cor = transformed_data.corr()
f, ax = plt.subplots(figsize=(6, 6))
cmap = sns.diverging_palette(0, 255, sep=9, n=256)
sns.heatmap(cor, cmap=cmap, linewidths=.001);<jupyter_output><empty_output><jupyter_text>As shown that all features exhibit weak correaltion with the target varaible.<jupyter_code>abs(cor.target).describe()<jupyter_output><empty_output><jupyter_text>Reducing dimentionality may improve the accuracy. Hence, here will remove some features that has very little correlation with the target.<jupyter_code>#remove the least correlated features
selected_data = transformed_data[cor.target[cor.target > 0.019].index.to_list()]
selected_data.head()
#feature importance using randomforest
x = selected_data.drop('target', axis=1)
y = selected_data.target
#split data for train and test
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.3, random_state=3, shuffle=True)

#fit the model
model = rfc()
model.fit(train_x, train_y)

feature_importance = pd.Series(model.feature_importances_, index=selected_data.drop('target', axis=1).columns)
feature_importance.nlargest(len(selected_data)).plot(kind='barh', alpha=.6, figsize=(9,12));<jupyter_output><empty_output><jupyter_text>From the barchart, most of the features exhibit certain degree of importance. It would be better to keep most of them.<jupyter_code>feature_importance.describe()
#recursive feature elimination with cross-validation using Random Forest to find the optimal number of features and the features
#feature selection can be done by setting a threshold if this step takes a lot of time
model = rfc()
rfecv = RFECV(estimator=model, step=1, cv=6, scoring='accuracy') #use 6-fold cross validation
rfecv = rfecv.fit(train_x, train_y)
optimal_features = train_x.columns[rfecv.support_]
print(len(optimal_features))
print(optimal_features)<jupyter_output>41
Index(['low', 'close', 'month', 'day_sin', 'day_cos', 'inphase', 'maxindex',
       'minindex', 'minidx', 'sum', 'cos', 'ln', 'sqrt', 'minus_di',
       'minus_dm', 'plus_dm', 'upperband', 'middleband', 'lowerband', 'dema',
       'ema', 'ht_trendline', 'kama', 'ma', 'mama', 'fama', 'midprice', 'sar',
       'sma', 't3', 'tema', 'trima', 'wma', 'avgprice', 'wclprice',
       'linearreg', 'linearreg_intercept', 'stddev', 'tsf', 'ad', 'obv'],
      dtype='object')<jupyter_text>As observed that, the optimal features selected by recursive feature elimination with cross validation are those that exhibits high importance on the feature importance barchart.<jupyter_code>#remove features with very little importance
# selected_data = selected_data[feature_importance[feature_importance > 0.003].index.to_list()]
#choose features from recursive cross validation
selected_data = selected_data[optimal_features]
selected_data.head()
#visualize the final data before machine learning
selected_data.plot(style='.', subplots=True, legend=False, use_index=False, alpha=0.3,
                title=list(selected_data.columns), layout=(5,9), figsize=(12,12)); #colormap="winter"<jupyter_output><empty_output><jupyter_text>As mentioned, feature selection has successfully remove all the trivial features and some unimportant features. There are still a small numer of 'outliers' being trained, however, using Random Forest.<jupyter_code>#prepare train and test sets
final_data = selected_data
x = final_data
y = np.where(transformed_data.close.shift(-5) > transformed_data.close, 1, -1)
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.3, random_state=3, shuffle=True)<jupyter_output><empty_output><jupyter_text>The train and test data are splitted randomly from the original dataset. The time-series data can be randomly splitted since the date information and its trigonometry functions were computed and made to be the independent variables.<jupyter_code>#tune hyperparameters to get highest accuracy
#
accuracy = []
parameter = np.linspace(200, 250, num=50, dtype='int')
for n in parameter:
    model = rfc(n_estimators=n, max_depth=98)
    model.fit(train_x, train_y)
    prediction = model.predict(test_x)
    accuracy.append(model.score(test_x, test_y))
pd.Series(accuracy,parameter).plot()
print(pd.Series(accuracy,parameter).idxmax())
#perform prediction and show report
#default parameter also yield good accuracy
model = rfc(n_estimators=234, max_depth=98)
model.fit(train_x, train_y)
prediction = model.predict(test_x)
sns.heatmap(confusion_matrix(test_y, prediction), cmap='viridis', annot=True, fmt='d');
print(classification_report(test_y, prediction))<jupyter_output>precision    recall  f1-score   support

          -1       0.80      0.73      0.76       897
           1       0.82      0.87      0.84      1263

    accuracy                           0.81      2160
   macro avg       0.81      0.80      0.80      2160
weighted avg       0.81      0.81      0.81      2160<jupyter_text>The 81% test accuracy looks promising. The model also exhibits high precision. However, the recall of '-1' is 72% which is 14% lower than the recall of '1'. This means that although the algorithm exhibits high accuracy and precision in predicting rise and decline of the market, it lacks high sensitivity of predicting decline. It is expected as there was some bias within the dataset as mentioned. To minimize the bias, resampling can be performed. However, how well the resampling on such dataset will need to be investigated. If a safer stock investment approach is preferred, an algorithm with higher sensitivity to decline is highly required.<jupyter_code>#save the model
#load the model and predict the whole dataset
#the new accuracy should be 0.7+0.3*previous accuracy
joblib.dump(model, "stock_movement_rfc.pkl") 
loaded_model = joblib.load("stock_movement_rfc.pkl")
new_prediction = loaded_model.predict(final_data)
print(classification_report(y, new_prediction))<jupyter_output>precision    recall  f1-score   support

          -1       0.94      0.92      0.93      2975
           1       0.94      0.96      0.95      4222

    accuracy                           0.94      7197
   macro avg       0.94      0.94      0.94      7197
weighted avg       0.94      0.94      0.94      7197
{'fraction_non_alphanumeric': 0.10190025724006306, 'fraction_numerical': 0.022072857024313334, 'mean_word_length': 5.130213631739573, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '477795', 'license': 'MIT', 'n_tokens_mistral': 3824, 'n_tokens_neox': 3626, 'n_words': 1145, 'path': 'stock_movement_classification.ipynb', 'repo_name': 'syl1618/Stock-Movement-Classification', 'size': '12051'}