<jupyter_start><jupyter_text>Info* Main Dataset: [S&P 500 stock data](https://www.kaggle.com/camnugent/sandp500)* Download detailes for each company: [S&P 500 Companies with Financial Information](https://datahub.io/core/s-and-p-500-companies-financialsresource-s-and-p-500-companies-financials_zip)Stock prices are flutuated in every day. So, in each day, put those stocks in order of price change to one sentence. Then, with certain window size, each stock will show up with highly related stock frequently, because they tend to move their prices together. Source: [stock2vec repo](https://github.com/kh-kim/stock2vec)  Imports<jupyter_code>import sys
import pandas as pd
import operator
import numpy as np
from gensim.models import Word2Vec
from sklearn.decomposition import PCA
from matplotlib import pyplot
import matplotlib.patches as mpatches
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.ensemble import RandomForestClassifier
from sklearn import svm, tree
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.datasets import make_classification
from sklearn.naive_bayes import GaussianNB
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Helper Functions<jupyter_code>def sort_dict(mydict, reversed=False):
  return sorted(mydict.items(), key=operator.itemgetter(1), reverse=reversed)<jupyter_output><empty_output><jupyter_text>Read Data<jupyter_code># companies' description
desc_df = pd.read_csv('../data/constituents.csv')
print('\nCompanies Details')
print(desc_df.head())

# stocks details
stocks_df = pd.read_csv('../data/all_stocks_5yr.csv')#, parse_dates=['date'])
print('\nCompanies Stocks')
print(stocks_df.head())<jupyter_output>Companies Details
  Symbol                 Name                  Sector
0    MMM           3M Company             Industrials
1    AOS      A.O. Smith Corp             Industrials
2    ABT  Abbott Laboratories             Health Care
3   ABBV          AbbVie Inc.             Health Care
4    ACN        Accenture plc  Information Technology

Companies Stocks
         date   open   high    low  close    volume Name
0  2013-02-08  15.07  15.12  14.63  14.75   8407500  AAL
1  2013-02-11  14.89  15.01  14.26  14.46   8882000  AAL
2  2013-02-12  14.45  14.51  14.10  14.27   8126000  AAL
3  2013-02-13  14.30  14.94  14.25  14.66  10259500  AAL
4  2013-02-14  14.94  14.96  13.16  13.99  31879900  AAL<jupyter_text>Preprocess<jupyter_code># dicitonary for companies name and sector
companies_names = {symbol:name for symbol, name in desc_df[['Symbol', 'Name']].values}
companies_sector = {symbol:sector for symbol, sector in desc_df[['Symbol', 'Sector']].values}

# get all companies symbols
symbols = stocks_df['Name'].values
dates = set(stocks_df['date'].values)
dates = sorted(dates)

# store each individual date and all its stocks
dates_dictionary = {date:{} for date in dates}<jupyter_output><empty_output><jupyter_text>Data for Word EmbeddingsFor each date in out dataset we rearrange each company in ascending order based on the **change in price**.Formula for **change in price** [source](https://pocketsense.com/calculate-market-price-change-common-stock-4829.html):* (closing_price - opening_price) / opening_priceWe can change the formula to use highest price and lowest price. This is something we will test out.<jupyter_code># calculate price change for each stock and sort them in each day
for date, symbol, op, cl, in stocks_df[['date', 'Name', 'open', 'close']].values:
  # CHANGE IN PRICE: (closing_price - opening_price) / opening_price
  dates_dictionary[date][symbol] = (cl - op)/op
# sort each day reverse order
dates_dictionary = {date:sort_dict(dates_dictionary[date]) for date in dates}

stocks_w2v_data = [[value[0] for value in dates_dictionary[date]] for date in dates]

# print sample
print(stocks_w2v_data[0])<jupyter_output>['MCO', 'MNST', 'SPGI', 'JNPR', 'AAL', 'BBY', 'INTU', 'SRCL', 'SCHW', 'MCHP', 'FLR', 'CL', 'ILMN', 'PVH', 'FB', 'M', 'IRM', 'VAR', 'DAL', 'BA', 'IT', 'BAC', 'EXC', 'ETR', 'XRX', 'O', 'LEN', 'LB', 'KLAC', 'PWR', 'RJF', 'HUM', 'C', 'VFC', 'EL', 'GLW', 'DHI', 'NEM', 'AEE', 'RMD', 'PG', 'RHT', 'RHI', 'MAS', 'EFX', 'DPS', 'IVZ', 'KSU', 'AES', 'NFLX', 'AXP', 'SIG', 'MU', 'TDG', 'RF', 'HIG', 'FDX', 'VZ', 'IDXX', 'PNC', 'T', 'LUK', 'ABBV', 'TRV', 'DVA', 'KMI', 'CTSH', 'CRM', 'FCX', 'ADM', 'PFE', 'CTAS', 'AMG', 'EQT', 'CCL', 'DGX', 'AKAM', 'NEE', 'GT', 'PEP', 'GPS', 'HCA', 'KO', 'NFX', 'COF', 'PDCO', 'BF.B', 'LEG', 'MET', 'SWK', 'NLSN', 'HRS', 'MDLZ', 'ARE', 'PEG', 'HP', 'CMS', 'ICE', 'DRI', 'MYL', 'SO', 'KMB', 'AJG', 'GRMN', 'DFS', 'BBT', 'CLX', 'PAYX', 'AFL', 'ETN', 'MKC', 'CSCO', 'NRG', 'ANSS', 'UAA', 'NI', 'KORS', 'K', 'TIF', 'UTX', 'BRK.B', 'DLR', 'F', 'GE', 'NVDA', 'NWL', 'EMR', 'A', 'ES', 'AIZ', 'PPL', 'NKE', 'JEC', 'AEP', 'DTE', 'SEE', 'ED', 'ABT', 'WY', 'HSIC', 'WU', 'PCG', 'RTN', 'QCO[...]<jupyter_text>ClassifiersWe utilized classifiers to predict a stock's sector based on its daily price change to determine the quality of the Word2Vec embedding and optimal number of dimensions. After testing 4 classifiers for up to 30 dimensions, we found that the accuracy of our classifiers begins to plateau at 4 dimensions.<jupyter_code>stocks_ordered = [[value[0] for value in dates_dictionary[date]] for date in dates]
featureNumber = 15
labels = ['Industrials' ,'Health Care' ,'Information Technology' ,'Utilities','Financials','Materials', 
                     'Consumer Discretionary','Real Estate', 'Consumer Staples','Energy',
                     'Telecommunication Services']
counter = []
classifier1_array = []
classifier2_array = []
classifier3_array = []
classifier4_array = []

# test accuracy at various number of dimensions
for j in range(1,21):
    model = Word2Vec(stocks_w2v_data, min_count=1, size=j)
    words = list(model.wv.vocab)
    X = model[model.wv.vocab]
    Y = list()
    for word in words:
        Y.append(companies_sector[word])

    # split data for cross validation
    X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)
    
    # predict sectors using GaussianNB, SVM, DecisionTreeClassifier and RandomForestClassifier
    model1 = GaussianNB()
    model1.fit(X_train, Y_train)
    preds1 = model1.predict(X_test)
    
    model2 = svm.SVC()
    model2.fit(X_train, Y_train)
    preds2 = model2.predict(X_test)
    
    model3 = tree.DecisionTreeClassifier()
    model3.fit(X_train, Y_train)
    preds3 = model3.predict(X_test)
    
    model4 = RandomForestClassifier()
    model4.fit(X_train, Y_train)
    preds4 = model4.predict(X_test)
    
    classifier1_array.append(accuracy_score(Y_test, preds1))
    classifier2_array.append(accuracy_score(Y_test, preds2))
    classifier3_array.append(accuracy_score(Y_test, preds3))
    classifier4_array.append(accuracy_score(Y_test, preds4))
    
    counter.append(j)

np.set_printoptions(threshold=sys.maxsize)

pyplot.plot(counter,classifier1_array)
pyplot.plot(counter,classifier2_array)
pyplot.plot(counter,classifier3_array)
pyplot.plot(counter,classifier4_array)
pyplot.ylabel('Accuracy')
pyplot.xlabel('Dimensions')
gnb_patch=mpatches.Patch(color='blue', label='GaussianNB')
svm_patch=mpatches.Patch(color='orange', label='svm')
dtc_patch=mpatches.Patch(color='green', label='Decision Tree')
rfc_patch=mpatches.Patch(color='red', label='Random Forest')
pyplot.legend(handles=[gnb_patch,svm_patch, dtc_patch, rfc_patch], loc='best')
pyplot.show()<jupyter_output>C:\Users\phill\Anaconda3\lib\site-packages\ipykernel_launcher.py:15: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).
  from ipykernel import kernelapp as app
C:\Users\phill\Anaconda3\lib\site-packages\sklearn\svm\base.py:193: FutureWarning: The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.
  "avoid this warning.", FutureWarning)
C:\Users\phill\Anaconda3\lib\site-packages\sklearn\ensemble\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.
  "10 in version 0.20 to 100 in 0.22.", FutureWarning)
C:\Users\phill\Anaconda3\lib\site-packages\ipykernel_launcher.py:15: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).
  from ipykernel import kernelapp as [...]<jupyter_text>Plot 4D RepresentationEach stock is now represented in 4 dimensions, or four numbers. To visualize these dimensions, we graphed the 4 dimensions in two separate 2D plots.<jupyter_code># confusion matrix function
def plot_cm(y_true, y_pred, class_names):
    cm = confusion_matrix(y_true, y_pred)
    fig, ax = pyplot.subplots(figsize=(20, 15)) 
    ax = sns.heatmap(
        cm, 
        annot=True, 
        fmt="d", 
        cmap=sns.diverging_palette(230, 30, n=9),
        ax=ax,
        annot_kws={"fontsize":20}
    )
    ax.xaxis.set_ticks_position('top')
    ax.xaxis.set_label_position('top')
    pyplot.ylabel('Actual', fontsize = 20)
    pyplot.xlabel('Predicted', fontsize = 20)
    ax.set_title('Confusion Matrix', fontsize = 40, y = -.02)
    ax.set_xticklabels(class_names, fontsize=20, rotation=90)
    ax.set_yticklabels(class_names, rotation=0, fontsize = 20)
    b, t = pyplot.ylim() # discover the values for bottom and top
    b += 0.5 # Add 0.5 to the bottom
    t -= 0.5 # Subtract 0.5 from the top
    pyplot.ylim(b, t) # update the ylim(bottom, top) values
    pyplot.show()
# recreate model with 4 dimensions(this is the model that will be used for the rest of the code)
model = Word2Vec(stocks_w2v_data, min_count=1, size=4)
words = list(model.wv.vocab)
X = model[model.wv.vocab]
Y = list()
for word in words:
    Y.append(companies_sector[word])

# split data set for cross validation
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)
# plot confusion matrix for SVM classifier at 4 dimensions
model2.fit(X_train, Y_train)
preds2= model2.predict(X_test)
acc = accuracy_score(Y_test, preds2)
print("Accuracy is ", acc)
plot_cm(Y_test,preds2,labels)
array1X = []
array1Y = []
array2X = []
array2Y = []

# color code by sector
sector_color_dict = {'Industrials':'red','Health Care':'orange','Information Technology':'yellow','Utilities':'green',
                     'Financials':'blue','Materials':'purple','Consumer Discretionary':'cyan','Real Estate':'magenta',
                     'Consumer Staples':'pink','Energy':'brown','Telecommunication Services':'gray'}
cvec = [sector_color_dict[companies_sector[word]] for word in words]

# legend
red_patch=mpatches.Patch(color='red', label='Industrials')
orange_patch=mpatches.Patch(color='orange', label='Health Care')
yellow_patch=mpatches.Patch(color='yellow', label='Information Technology')
green_patch=mpatches.Patch(color='green', label='Utilities')
blue_patch=mpatches.Patch(color='blue', label='Financials')
purple_patch=mpatches.Patch(color='purple', label='Materials')
cyan_patch=mpatches.Patch(color='cyan', label='Consumer Discretionary')
magenta_patch=mpatches.Patch(color='magenta', label='Real Estate')
pink_patch=mpatches.Patch(color='pink', label='Consumer Staples')
brown_patch=mpatches.Patch(color='brown', label='Energy')
gray_patch=mpatches.Patch(color='gray', label='Telecommunication Services')

# graph dimensions 1 & 2
for d in range(0,505):
    array1X.append((model[model.wv.vocab])[d,0])
    array1Y.append((model[model.wv.vocab])[d,1])
pyplot.figure(num=None, figsize=(14, 8), dpi=80, facecolor='w', edgecolor='k')
pyplot.scatter(array1X[:], array1Y[:], c=cvec)
pyplot.legend(handles=[red_patch,orange_patch,yellow_patch,green_patch,blue_patch,purple_patch,cyan_patch,magenta_patch,
                       pink_patch,brown_patch,gray_patch],loc='best')
pyplot.show()

# graph dimensions 3 & 4
for f in range(0,505):
    array2X.append((model[model.wv.vocab])[f,2])
    array2Y.append((model[model.wv.vocab])[f,3])
pyplot.figure(num=None, figsize=(14, 8), dpi=80, facecolor='w', edgecolor='k')
pyplot.scatter(array2X[:], array2Y[:], c=cvec)
pyplot.legend(handles=[red_patch,orange_patch,yellow_patch,green_patch,blue_patch,purple_patch,cyan_patch,magenta_patch,
                       pink_patch,brown_patch,gray_patch],loc='best')
pyplot.show()<jupyter_output>C:\Users\phill\Anaconda3\lib\site-packages\ipykernel_launcher.py:27: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).
C:\Users\phill\Anaconda3\lib\site-packages\ipykernel_launcher.py:28: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).<jupyter_text>Train Word EmbeddingsWe're using PCA dimensionality reduction to reduce our 4D model into 2 dimensions that can easily be graphed on a coordinate plane.<jupyter_code># fit a 2D PCA model to the vectors
X = model[model.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
newResultX = []
newResultY = []
newWords = list()
newWordToken = ""
# read stock names from an input file (format: %AAL%)
with open('../data/stocks.txt') as stockFile:
    contents = stockFile.read()
    for i in range(0,505):
        newWordToken = "%" + words[i] + "%"
        #if newWordToken in contents:
        newWords.append(words[i])
        newResultX.append(result[i,0])
        newResultY.append(result[i,1])<jupyter_output>C:\Users\phill\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).<jupyter_text>Plot 2D Representation<jupyter_code>#increase size of figure
pyplot.figure(num=None, figsize=(14, 8), dpi=80, facecolor='w', edgecolor='k')

# color code by sector
sector_color_dict = {'Industrials':'red','Health Care':'orange','Information Technology':'yellow','Utilities':'green',
                     'Financials':'blue','Materials':'purple','Consumer Discretionary':'cyan','Real Estate':'magenta',
                     'Consumer Staples':'pink','Energy':'brown','Telecommunication Services':'gray'}
cvec = [sector_color_dict[companies_sector[word]] for word in newWords]

# create a scatter plot of the projection
pyplot.scatter(newResultX[:], newResultY[:], c = cvec)

# labels(company symbol)
'''
for i, word in enumerate(newWords):
	#pyplot.annotate(companies_names[word], xy=(newResultX[i], newResultY[i]), fontsize = 12)
    #pyplot.annotate(word, xy=(newResultX[i], newResultY[i]))
'''

pyplot.legend(handles=[red_patch,orange_patch,yellow_patch,green_patch,blue_patch,purple_patch,cyan_patch,magenta_patch,
                       pink_patch,brown_patch,gray_patch],loc='best')
pyplot.show()

# display eigen pairs(coordinate points)
print(result)
pca.fit(X)
eigen_vals = pca.explained_variance_
eigen_vecs = pca.components_
eigen_pairs = [(np.abs(eigen_vals[i]), eigen_vecs[:,i])for i in range(len(eigen_vals))]
eigen_pairs.sort(key=lambda k: k[0], reverse=True)
w = np.hstack((eigen_pairs[0][1][:,np.newaxis],eigen_pairs[1][1][:,np.newaxis]))
print('Matrix W;\n',w)<jupyter_output><empty_output><jupyter_text>Analyze Features<jupyter_code># compute average feature value for each sector(15 total features)
np.set_printoptions(threshold=sys.maxsize)

# output info to averageFeatures file
with open("../notebooks/averageFeatures.txt", 'w') as averageFile:
    for k in range(0, len(sectors)):
        companiesInSector = 0
        averages = []
        for i in range (0, featureNumber):
            averages.append(0.0)
        for i in range(0,505):
            if companies_sector[words[i]] == sectors[k]:
                companiesInSector += 1
                for j in range(0,featureNumber):
                    averages[j] += model[model.wv.vocab][i,j]
        for i in range (0,featureNumber):
            averages[i] /= companiesInSector;
        averageFile.write(sectors[k])
        averageFile.write(" Average Feature Numbers: ")
        averageFile.write("\n")
        for i in range(0, featureNumber):
            averageFile.write(str(averages[i]) + " ")
    
        averageFile.write("\n\n")
# print similar stocks
target_symb = 'ALXN'

print('Symbol:%s\tName:%s\tSector: %s'%(target_symb, companies_names[target_symb], companies_sector[target_symb]))
top_similar = model.similar_by_word(target_symb, topn=20)
print('Most Similar')
for similar in top_similar:
  symb = similar[0]
  name = companies_names[symb]
  sect = companies_sector[symb]
  print('Symbol: %s\tName: %s\t\t\tSector: %s'%(symb, name, sect))
# access vector for one word
print(model['AAL'])<jupyter_output>[ 0.35754925 -0.92679065  0.2502123   0.45728815 -0.58478105  2.534188
  0.20739758 -0.16268525 -0.32791242 -0.87240684 -0.7866834  -0.1181294
 -1.032652    0.04860113  0.5716841   1.8690357  -0.97949237  0.34410962
  0.16535728  0.33681223]
{'fraction_non_alphanumeric': 0.14143496701225258, 'fraction_numerical': 0.03605089538171536, 'mean_word_length': 4.834020618556701, 'pattern_counts': {'":': 1, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '420725', 'license': 'MIT', 'n_tokens_mistral': 6451, 'n_tokens_neox': 5941, 'n_words': 1400, 'path': 'train_embeddings-checkpoint.ipynb', 'repo_name': 'yuvi18/stock_risk_prediction', 'size': '16976'}