<jupyter_start><jupyter_text>Chapter 4 Question 10A comparison of logistic regression, LDA, QDA and KNN on stock market data<jupyter_code>import statsmodels.api as sm
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.graphics.tsaplots
import numpy as np
import sklearn.linear_model
import sklearn.metrics
import sklearn.discriminant_analysis
import sklearn.neighbors
import pandas as pd

sns.set(style="whitegrid")
# This dataset contains weekly returns for 21 years, from 1990 to 2010.
weekly = sm.datasets.get_rdataset("Weekly", "ISLR").data<jupyter_output><empty_output><jupyter_text>(a) Produce some numerical and graphical summaries of the `weekly` dataset<jupyter_code>weekly.describe()
sns.pairplot(weekly)
print(f"Market goes up on average {sum(weekly.Direction=='Up')/len(weekly)*100:.2f}% of the time")<jupyter_output>Market goes up on average 55.56% of the time<jupyter_text>Patterns: - Volume goes steeply up with time - Nothing else obvious<jupyter_code># Convert this to a time series and do some averages / plots of total change / downsampling?
plt.plot(weekly.Today)
# Autocorrelation of the "Today" data
# We don't really expect significant correlation
fig = statsmodels.graphics.tsaplots.plot_acf(weekly.Today, lags=25, title="Autocorrelation in weekly stock prices")<jupyter_output><empty_output><jupyter_text>(b) Use the full data set to perform a logistic regression with `Direction` as the response and the five lag variables plus `Volume` as the predictors.<jupyter_code># Could use sklearn.linear_model.LogisticRegression
y = weekly.Direction=="Up"  # Have to encode this ourselves, apparently. 
X = sm.add_constant(weekly.drop(columns=["Direction","Year","Today"]))
logistic_model =  sm.Logit(y, X)
results = logistic_model.fit()
print(results.summary())
model = sklearn.linear_model.LogisticRegression()
model.fit(weekly.drop(columns=["Direction","Year","Today"]), weekly.Direction)

print(model.intercept_)
print(model.coef_)<jupyter_output>[0.26484745]
[[-0.04117292  0.05846974 -0.01599122 -0.02769998 -0.01440289 -0.02212844]]<jupyter_text>The two regressors agree, but only the intercept is significant (i.e. there is no-zero probability of "Up" when the lags and volume are all zero). The `Lag2` is significant at a p<0.05 level but needs further investigation, with the Bonferroni correction it wouldn't be.  (c) compute the confusion matrix and overall fraction of correct predictions<jupyter_code>y_true = weekly.Direction
y_pred = model.predict(weekly.drop(columns=["Direction","Year","Today"]))

confusion_matrix = sklearn.metrics.confusion_matrix(y_true, y_pred)
tn, fp, fn, tp = confusion_matrix.ravel()
print(confusion_matrix)
fraction_correct = (tn+tp)/(tn+tp+fn+fp)
print(f"fraction correct: {fraction_correct}")<jupyter_output>[[ 55 429]
 [ 47 558]]
fraction correct: 0.5629017447199265<jupyter_text>So we are right 56.3% of the time, versus a model that always predicts "Up" which would be right 55.6% of the time. We don't have many false negatives, but we have an awful lot of false positives!<jupyter_code># Check for statsmodels version
y_pred_sm = np.where(results.fittedvalues < 0.5, "Up", "Down")
print(sklearn.metrics.confusion_matrix(y_true, y_pred_sm))<jupyter_output>[[ 19 465]
 [ 42 563]]<jupyter_text>(d) Now fit the model using training data from 1990-2008, with `Lag2` as the only predictor. Use the 2009-2010 data as test data, and get the confusion matrix & overall fraction correct.<jupyter_code>train = weekly[weekly.Year < 2009]
test = weekly[weekly.Year >= 2009]
x_train = train.Lag2
y_train = train.Direction
x_test = test.Lag2
y_test = test.Direction
Ky_pred = model.predict(x_test.to_numpy().reshape(-1,1))
confusion_matrix = sklearn.metrics.confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = confusion_matrix.ravel()
print(confusion_matrix)
fraction_correct = (tn+tp)/(tn+tp+fn+fp)
print(f"fraction correct: {fraction_correct}")<jupyter_output>[[ 9 34]
 [ 5 56]]
fraction correct: 0.625<jupyter_text>(e) Repeat (d) using LDA<jupyter_code>lda_model = sklearn.discriminant_analysis.LinearDiscriminantAnalysis()
lda_model.fit(x_train.to_numpy().reshape(-1,1), y_train) # reshape required to cast the training data to a 2d array
y_pred = lda_model.predict(x_test.to_numpy().reshape(-1,1))

confusion_matrix = sklearn.metrics.confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = confusion_matrix.ravel()
print(confusion_matrix)
fraction_correct = (tn+tp)/(tn+tp+fn+fp)
print(f"fraction correct: {fraction_correct}")<jupyter_output>[[ 9 34]
 [ 5 56]]
fraction correct: 0.625<jupyter_text>(f) Repeat (d) using QDA<jupyter_code>qda_model = sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis()
qda_model.fit(x_train.to_numpy().reshape(-1,1), y_train) # reshape required to cast the training data to a 2d array
y_pred = qda_model.predict(x_test.to_numpy().reshape(-1,1))

confusion_matrix = sklearn.metrics.confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = confusion_matrix.ravel()
print(confusion_matrix)
fraction_correct = (tn+tp)/(tn+tp+fn+fp)
print(f"fraction correct: {fraction_correct}")<jupyter_output>[[ 0 43]
 [ 0 61]]
fraction correct: 0.5865384615384616<jupyter_text>(g) Repeat (d) using KNN with K = 1<jupyter_code>knn_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=1)
knn_model.fit(x_train.to_numpy().reshape(-1,1), y_train) # reshape required to cast the training data to a 2d array
y_pred = knn_model.predict(x_test.to_numpy().reshape(-1,1))

confusion_matrix = sklearn.metrics.confusion_matrix(y_test, y_pred)
tn, fp, fn, tp = confusion_matrix.ravel()
print(confusion_matrix)
fraction_correct = (tn+tp)/(tn+tp+fn+fp)
print(f"fraction correct: {fraction_correct}")<jupyter_output>[[21 22]
 [31 30]]
fraction correct: 0.49038461538461536<jupyter_text>(h) Which of these methods appears to provide the best results on this data? Somewhat surprisingly, it seems like a linear fit performs best on this data, better than QDA and KNN (insinuating that both classes have the same variance in `Lag2`)  (i) Experiment with different combinations of predictors, including possible transformations and interactions, for each of the methods. Report the best results and parameters.<jupyter_code># The p-value for ShelveLoc suggests it is predictive - plot with FacetGrid
# fig, ax = plt.subplots(figsize=(10,10))

g = sns.FacetGrid(weekly, hue="Direction", height=5)
g = g.map(sns.distplot, "Lag2").add_legend()


import itertools

for i in range(1,len(train.columns.drop(["Direction","Year","Today"]))):
    for col in itertools.combinations(train.columns.drop(["Direction","Year","Today"]), i):
        col = list(col)
        print(col)
        x_train = train[col]
        y_train = train.Direction
        x_test = test[col]
        y_test = test.Direction

        lda_model = sklearn.discriminant_analysis.LinearDiscriminantAnalysis()
        lda_model.fit(x_train, y_train) # reshape required to cast the training data to a 2d array
        y_pred = lda_model.predict(x_test)

        confusion_matrix = sklearn.metrics.confusion_matrix(y_test, y_pred)
        tn, fp, fn, tp = confusion_matrix.ravel()
        print(confusion_matrix)
        fraction_correct = (tn+tp)/(tn+tp+fn+fp)
        print(f"fraction correct: {fraction_correct}")
# With `Lag2`, we get an accuracy of 0.625. With other predictors, the data gets worse. What about with QDA?
for i in range(1,len(train.columns.drop(["Direction","Year","Today"]))):
    for col in itertools.combinations(train.columns.drop(["Direction","Year","Today"]), i):
        col = list(col)
        print(col)
        x_train = train[col]
        y_train = train.Direction
        x_test = test[col]
        y_test = test.Direction

        qda_model = sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis()
        qda_model.fit(x_train, y_train) # reshape required to cast the training data to a 2d array
        y_pred = qda_model.predict(x_test)

        confusion_matrix = sklearn.metrics.confusion_matrix(y_test, y_pred)
        tn, fp, fn, tp = confusion_matrix.ravel()
#         print(confusion_matrix)
        fraction_correct = (tn+tp)/(tn+tp+fn+fp)
        print(f"fraction correct: {fraction_correct}")

# QDA strictly worse. And KNN?
for i in range(2,len(train.columns.drop(["Direction","Year","Today"]))):
    for col in itertools.combinations(train.columns.drop(["Direction","Year","Today"]), i):
        col = list(col)
        print(col)
        x_train = train[col]
        y_train = train.Direction
        x_test = test[col]
        y_test = test.Direction
        for j in range(1,6):
            print(j)
            knn_model = sklearn.neighbors.KNeighborsClassifier(n_neighbors=j)
            knn_model.fit(x_train, y_train) # reshape required to cast the training data to a 2d array
            y_pred = knn_model.predict(x_test)

            confusion_matrix = sklearn.metrics.confusion_matrix(y_test, y_pred)
            tn, fp, fn, tp = confusion_matrix.ravel()
    #         print(confusion_matrix)
            fraction_correct = (tn+tp)/(tn+tp+fn+fp)
            print(f"fraction correct: {fraction_correct}")

# And with interactions? transforms?<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11547268563951586, 'fraction_numerical': 0.029440628066732092, 'mean_word_length': 5.050131926121372, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '292646', 'license': 'MIT', 'n_tokens_mistral': 3329, 'n_tokens_neox': 3041, 'n_words': 783, 'path': 'Chapter4/question10.ipynb', 'repo_name': 'wllgrnt/islr-python-examples', 'size': '9171'}