<jupyter_start><jupyter_text>2. Clustering Indian Stocks using KMeans @techietraderThe idea is to try finding those stocks which belong to same clusters irrespective of the sector they belong. Once we find these clusters we find those stocks which are closest in distance (More on this later on) within the cluster to be later used for Pairs trading.Here is the Flow-1. Collection of Data and preprocessing. We are using Companies' financial performance data and also historical data to find proximity among them. We take two years daily closing data for the period 2016-18.  2. We cluster stocks using KMeans Clustering Algorithm. The idea is to cover as many stocks as we can in the whole stocks gamut in India. This we can achieve by closing down the gap between Total Market Capitalization of stocks and the cumulative Market Capitalization of all the stocks that we are considering.3. We than find those stocks inside the clusters which are closest in distance to each other using the measure called Euclidean Distance (More on this later).4. We than run either Augmented Dickey Fuller Test or Johansen Test to figure out whether the pairs are cointegrated. We use the same historical period as above to figure that out.5. Once we have the paris where the spread between them is stationary, we than backtest these pairs to figure out results.6. Optimization to improve resultsPart 2 & 3 above will be covered in this file.  PrerequisitesIn order to run this book on your local machine you require -1. Python2 (Anaconda Package)- https://www.anaconda.com/download/ .If you can manage to download individual packages on your own that should be fine too.2. Seaborn. - https://seaborn.pydata.org/ .  For better visualization  About meI am an Individual trader who loves to try researching new strategies. I m not associated with anyone mentioned in prerequisites above<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import sklearn as sk

print pd.__version__
print np.__version__
print sns.__version__
print sk.__version__<jupyter_output>0.20.3
1.15.0
0.7.1
0.19.0<jupyter_text>Import Libraries  Data Analysis<jupyter_code>import numpy as np
import pandas as pd
np.set_printoptions(suppress=True, formatter={'float_kind':'{:0.2f}'.format})
# Ignore any warnings that you might get while importing<jupyter_output><empty_output><jupyter_text>Visualization<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.manifold import TSNE  # Helps visualize higher dimensions data
import missingno as msno # Missing Data visualization

sns.set(style="white")
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Sklearn (Machine Learning framework)<jupyter_code>from sklearn.decomposition import PCA # Helps reduce dimensions and also solves problem of multicollinearity.
from sklearn.preprocessing import StandardScaler  # Helps us in Standardizing the data
from sklearn.cluster import KMeans
from sklearn.neighbors import NearestNeighbors<jupyter_output><empty_output><jupyter_text>Miscellaneous<jupyter_code>from datetime import datetime
from collections import Counter
import time<jupyter_output><empty_output><jupyter_text>Importing Files  Financial Data<jupyter_code>fin_data = pd.read_excel('Financial_data.xlsx', index_col= 'Ticker')<jupyter_output><empty_output><jupyter_text>Futures & Options List<jupyter_code>f_n_o_list = pd.read_excel('FnO_list.xlsx', index_col='Symbol')
f_n_o_list = list(f_n_o_list.index)
f_n_o_list = [scrip for scrip in f_n_o_list if scrip in fin_data.index]<jupyter_output><empty_output><jupyter_text>Returns' data<jupyter_code>returns_data = pd.read_excel('Returns_data.xlsx', index_col= 'Date')
returns_data.dropna(inplace= True)
# lets bring the dataframes we want to merge on same index
returns_data_t = returns_data.T.sort_index(ascending= True)
returns_data_t.index.name = 'Ticker'

fin_data = fin_data.sort_index(ascending= True)
print returns_data.T.shape
print fin_data.shape
# A small check whether all the rows match in the pricing and fin_data DFs
for i , j in zip(fin_data.index,returns_data_t.index):
    if i != j :
        print i, j
# Printed nothing
f_n_o_list[0:7]<jupyter_output><empty_output><jupyter_text>Lets look at the Market Capitalization we are covering  Marketcap of $2.29 Trillion<jupyter_code>Total_Market_Cap = 2.29 * 1000 * 1000 * 1000000 * 68

Makret_Cap_we_covering = fin_data['Market Cap'].sum() * 10000000

print 'We are covering {} % of Total Market Cap already'.format(round((Makret_Cap_we_covering / Total_Market_Cap)*100,2))<jupyter_output>We are covering 89.22 % of Total Market Cap already<jupyter_text>Great!The Market cap of $2.29 Trillion dates back to late 2017, whereas the financial data that we have is of recent date. The market cap that we shall be covering in our analysis should not be significantly less but less than overall Market Cap. Here is the article for reference - https://www.bloomberg.com/news/articles/2017-11-30/indian-stock-market-overtakes-canada-in-2-trillion-standoff  Time to move on with Machine Learning!<jupyter_code># Standardization. To bring everything on same scale
std = StandardScaler()
X_fin_data = pd.DataFrame(std.fit_transform(fin_data), index = fin_data.index, columns = fin_data.columns)
X_fin_data.head() #Standardized Financial Data
returns_data_t.head()
print X_fin_data.shape 
print returns_data_t.shape<jupyter_output>(968, 11)
(968, 492)<jupyter_text>Finding Principal Components for our tickers that explain highest Variance in dataset A very nice Step-by-Step guide to Principal Component Analysis is given by <NAME> on his youtube channel [here](https://www.youtube.com/watch?v=FgakZw6K1QQ)  Lets combine the above two Dfs to create our final input<jupyter_code>final_df = pd.merge(X_fin_data, returns_data_t, left_index= True, right_index= True)
final_df.shape # Merged Columns on same index
final_df.head()<jupyter_output><empty_output><jupyter_text>PCA<jupyter_code>pca = PCA(n_components= 7) # 7 PCs explains good variance
final_X = pca.fit_transform(final_df.values)
print pca.explained_variance_ratio_.sum() # 80% of variance is explained by 7 Principal componenets instead of 503.
print final_X.shape<jupyter_output>(968L, 7L)<jupyter_text>Tuning the Hyperparameters Elbow method is really nice way to find optimal value of K in kmeans clustering<jupyter_code>variance = []
for k in range(1, 75): # 125 is the number of sectors
    km = KMeans(n_clusters=k, 
                )
    km.fit(final_X)
    variance.append(km.inertia_)


plt.figure(figsize = (18,6))
plt.plot(range(1, 75), variance, marker='o')
plt.grid()
plt.xlabel('Number of clusters')
plt.ylabel('Variance')
plt.title('Elbow')<jupyter_output><empty_output><jupyter_text>Lets choose k as 40<jupyter_code>km = KMeans(n_clusters=40,init= 'random', max_iter= 800,n_init= 300,algorithm= 'elkan', random_state= 5)
km.fit(final_X)
df_clustered = pd.DataFrame(km.labels_, fin_data.index)

labels = km.labels_


clustered_series = pd.Series(index=fin_data.index, data=labels)
clustered_series.value_counts().head()<jupyter_output><empty_output><jupyter_text>t-distributed stochastic neighbor embedding (TSNE)Tsne is machine learning algorithm technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets.<jupyter_code># Ignore Warning

X_tsne = TSNE(learning_rate=1000, perplexity=25, random_state=1337).fit_transform(final_X)
X_tsne_df = pd.DataFrame(X_tsne, columns = ['X_tsne_1', 'X_tsne_2'])
X_tsne_df['labels'] = labels
X_tsne_df.head()
# Create the figure
fig = plt.figure( figsize=(14,8))
ax = fig.add_subplot(1, 1, 1, title='X-Tsne Kmeans' )
# Create the scatter
ax.scatter(
    x=X_tsne_df['X_tsne_1'], 
    y=X_tsne_df['X_tsne_2'], 
    c=X_tsne_df['labels'], 
    cmap=plt.cm.get_cmap('Paired'), 
    alpha=0.85,
    s=100)
# Number of stocks in each cluster
plt.figure(figsize= (16,6))
sns.countplot(x = clustered_series.index, data =clustered_series)
plt.xlabel = 'Number of Clusters'
plt.ylabel = 'Number of Stocks in each CLuster'<jupyter_output><empty_output><jupyter_text>Clustering helps us to figure out the stocks that are closest in distance to each other. It helps us reduce the brute force tries on all the pairs. Here is the example - Suppose we want to try a brute force method and we have 968 stocks, what we have to attempt in Brute Force technique is-<jupyter_code>print 'So total number of pairs we have to attempt is : {}'.format((
    len(returns_data.columns) * (len(returns_data.columns) -1)) /2)<jupyter_output>So total number of pairs we have to attempt is : 468028<jupyter_text>Thats a lot of combinations. This is where Clustering comes in handy. We only try conintegration on those many pairs of stocks that fall in same clusters, this not only reduces time and computational complexity but also gives accurate results<jupyter_code># Clusters Discovered
counts = clustered_series.value_counts()
ticker_count_reduced = counts[counts>1] # This eliminates clusters where their is only one stock
print "Clusters formed: %d" % len(ticker_count_reduced)
print "Pairs to evaluate: %d" % (ticker_count_reduced*(ticker_count_reduced-1)).sum()<jupyter_output>Clusters formed: 37
Pairs to evaluate: 49962<jupyter_text>But lets just stick to FnO listed Stocks and find number of pairs to be tested for cointegration<jupyter_code>relevant_stocks = [stock for stock in f_n_o_list if stock in returns_data.columns]
print 'Total number of stocks :- {}'.format(len(relevant_stocks))
print '\n'
print 'So total number of pairs we have to attempt for cointegration in brute force is = {}'.format((
    len(relevant_stocks) * (len(relevant_stocks) -1)) /2)
clusters_f_n_o = clustered_series[f_n_o_list]
# Clusters Discovered for FnO Segment
clusters_values = clusters_f_n_o.value_counts()
clusters_with_atleast_2_stocks = clusters_values[clusters_values > 1] # This eliminates clusters which less than 2 stocks in it
print 'Clusters formed: {}'.format(len(clusters_with_atleast_2_stocks))
print 'Pairs to evaluate: {}'.format((clusters_with_atleast_2_stocks*(clusters_with_atleast_2_stocks-1)).sum())
clusters_f_n_o.head() # Its a series with stock symbol and the cluster it belongs to<jupyter_output><empty_output><jupyter_text>Woow! We reduced attempts from 19306 to mere 2558  Lets try and check accuracy of our clusters using the Sector column. As discussed prior, we want stocks in same sector to fall in same cluster<jupyter_code>fin_data['Sub-Sector'].value_counts().head() # Index is sector number and the value is the number of companies in that
                                             # sector<jupyter_output><empty_output><jupyter_text>30th mapped sector has 52 companies. Lets just check accuracy<jupyter_code>dense_sector = list(fin_data[fin_data['Sub-Sector'] == 30].index)
print dense_sector[0:6], len(dense_sector)
print clustered_series[dense_sector].head(10)
clustered_series[dense_sector].value_counts().max() / float(clustered_series[dense_sector].value_counts().sum()) *100<jupyter_output>Ticker
ALICON        14
ASAHIINDIA    14
AUTOAXLES     14
BANCOINDIA    14
BOSCHLTD      15
DYNAMATECH    22
FIEMIND       14
FMGOETZE      14
GABRIEL       14
GANDHITUBE    14
dtype: int32<jupyter_text>71.15% of the stock in that sector was accurately clustered  Lets check accuracy for FnO stocks and their sectors<jupyter_code>fin_data_sector = fin_data['Sub-Sector']
fin_data_sector = fin_data_sector[f_n_o_list]
# No. of unique sector for FnO stocks
fin_data_sector.nunique()
fin_data_sector.head(5)
# lets find accuracy for top 10 dense sector
company_sector =  []
top_10_dense_sector = fin_data_sector.value_counts().head(10)
for num in top_10_dense_sector.index:
    company_sector = list(fin_data_sector[fin_data_sector == num].index)
    print 'Total Number of companies in sector {} is {}'.format(int(num),len(company_sector))

    temp = clusters_f_n_o[company_sector].value_counts()
    print 'The accuracy in clustering stocks is {} %'.format(round(temp.max() / float(temp.sum()) *100),3)
    print '\n'<jupyter_output>Total Number of companies in sector 13 is 13
The accuracy in clustering stocks is 31.0 %


Total Number of companies in sector 7 is 12
The accuracy in clustering stocks is 50.0 %


Total Number of companies in sector 2 is 11
The accuracy in clustering stocks is 27.0 %


Total Number of companies in sector 0 is 9
The accuracy in clustering stocks is 44.0 %


Total Number of companies in sector 17 is 9
The accuracy in clustering stocks is 33.0 %


Total Number of companies in sector 11 is 9
The accuracy in clustering stocks is 33.0 %


Total Number of companies in sector 9 is 7
The accuracy in clustering stocks is 29.0 %


Total Number of companies in sector 1 is 6
The accuracy in clustering stocks is 17.0 %


Total Number of companies in sector 5 is 5
The accuracy in clustering stocks is 40.0 %


Total Number of companies in sector 26 is 5
The accuracy in clustering stocks is 40.0 %<jupyter_text>Figures are dismal  Now Lets plot a few stocks from a few clusters to check their price movements and relationFor this we will start plotting stocks by reindexing them to 0 to also find outperforming stocks within the cluster<jupyter_code>pricing_data = pd.read_excel('Pricing_data.xlsx', index_col= 'Date')
# Final number of clusters for FnO segment
clusters_with_atleast_2_stocks.head()
# Lets just randomly choose clusters to plot
clusters_plot = [39,29,37]

# Plotting

for num in clusters_plot:
    stocks = list(clusters_f_n_o[clusters_f_n_o==num].index)
    data = pricing_data[stocks].apply(lambda x: (x - x[0])/x[0] )
    data.plot(figsize=(16,6),grid=True, title='Stock Plot for Cluster {}'.format(num) )<jupyter_output><empty_output><jupyter_text>Seems a mix bag. Some are outperforming , some closesly follow each other, etc, etc, etc....  Final Step!Finding pairs from each 21 cluster which has highest proximity. In other words finding stocks that are closest in distance Lets just briefly look as to what we are doing below.Knearest neigbors algorithm uses same distance metric to find proximity among elements. We used it to figure out the closest neighbor to each stock in the list. Lets understand this by an example of 3 points on a number line- Lets say we have three elements in a cluster. KNN algorithm finds the elements nearest to every element using Euclidean distance depending upon the number of K value we specify. So in this example we will have -1.  A, B2.  B, C3.  C, BThough they all belong to same cluster we would be more interested in finding those items in a pair we are closest to each other. In this case B and C So the idea is first we find stock closest to each n every stock and then filter them out on basis of Futures and Options List<jupyter_code>NN =NearestNeighbors(n_neighbors=2 , metric= 'euclidean').fit(pd.DataFrame(final_X, index = final_df.index))
distances, indices = NN.kneighbors(pd.DataFrame(final_X, index = final_df.index), n_neighbors=2)
indices[0:5] # [0,1,2,3,4] are the indices of the stocks as shown below. 
             # [885,357,846,955,485] are the indices of stocks that are closest in distance to above indices
final_pairs_df = pd.DataFrame(clustered_series, columns= ['Cluster_number'])
final_pairs_df.head()
final_pairs_df = final_pairs_df.reset_index().join(
    pd.DataFrame(indices)[1]).set_index('Ticker') # Adds new column of the index of the stock closest to the index stock

final_pairs_df.rename(columns= {1 : 'Nearest_Neighbors_Index'},inplace= True) # Rename column

final_pairs_df['Nearest_Neighbor'] = final_pairs_df.index[final_pairs_df['Nearest_Neighbors_Index']] # Extracting name of 
                                                                                                 # stock using index values

final_pairs_df = final_pairs_df.loc[f_n_o_list] # Considering stocks only part of FnO List

final_pairs_df = final_pairs_df.reset_index().set_index(['Cluster_number', 'Ticker']).sort_index() # Multi Indexing

final_pairs_df.drop('Nearest_Neighbors_Index', axis = 1, inplace= True) # Not required now
final_pairs_df.head()
print final_pairs_df.shape<jupyter_output>(197, 1)<jupyter_text>The nearest neighbor column has stocks from non-FnO list. So lets drop those rows as well<jupyter_code>final_pairs_df = final_pairs_df[final_pairs_df['Nearest_Neighbor'].isin(f_n_o_list)]
final_pairs_df.head()
final_pairs_df.to_csv('Pairs.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07846715328467153, 'fraction_numerical': 0.02226277372262774, 'mean_word_length': 5.321030372933487, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 91, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '319328', 'license': 'MIT', 'n_tokens_mistral': 5535, 'n_tokens_neox': 5106, 'n_words': 1789, 'path': 'Clustering_Stocks_2.ipynb', 'repo_name': 'techietrader/Pairs-Trading-Indian-Stocks-using-Machine-Learning', 'size': '16446'}