<jupyter_start><jupyter_text>The Stock Market Data<jupyter_code>df.columns.tolist()
df.shape
df.describe()
df['Direction'].describe()
# pairwise correlations
df.corr()<jupyter_output><empty_output><jupyter_text>The correlations between the lag variables (previous days’) and today’s returns are close to zero.<jupyter_code>sns.boxplot(x=df['Year'], y=df['Volume'], palette=sns.color_palette("Blues", 1));<jupyter_output><empty_output><jupyter_text>The average number of shares traded daily increased from 2001 to 2005.  K-Nearest NeighborDirection = whether the market was Up or Down  Lag1, Lag2 = percentage returns for the two previous trading days  Predict Direction from Lag1 and Lag2:<jupyter_code>model = KNeighborsClassifier(n_neighbors=3)<jupyter_output><empty_output><jupyter_text>Model FitUse the observations from 2001 to 2004 as training data and the observations from 2005 as testing data.<jupyter_code>xtrain = df[df['Year'] != 2005][['Lag1', 'Lag2']]
xtest = df[df['Year'] == 2005][['Lag1', 'Lag2']]

ytrain = df[df['Year'] != 2005][['Direction']]
ytest = df[df['Year'] == 2005][['Direction']]

print xtrain.shape
print xtest.shape
model.fit(X=xtrain, y=ytrain)<jupyter_output>/Users/divyanair/.pyenv/versions/2.7.14/envs/interview_env/lib/python2.7/site-packages/ipykernel_launcher.py:1: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  """Entry point for launching an IPython kernel.<jupyter_text>Model Accuracy 1. Training Error<jupyter_code>ypredTrain = model.predict(X=xtrain)
ypredTrain[:5]
np.mean(np.equal(ypredTrain, ytrain['Direction'].tolist()))<jupyter_output><empty_output><jupyter_text>2. Testing Error<jupyter_code>ypred = model.predict(X=xtest)
ypred[:5]
confusion_matrix(ytest, ypred, labels=["Down", "Up"])
np.mean(np.equal(ypred, ytest['Direction'].tolist()))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14026915113871635, 'fraction_numerical': 0.030020703933747412, 'mean_word_length': 7.86697247706422, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '237745', 'license': 'MIT', 'n_tokens_mistral': 726, 'n_tokens_neox': 663, 'n_words': 150, 'path': 'labs/chapter4/KNN.ipynb', 'repo_name': 'dsnair/ISLR', 'size': '1932'}