<jupyter_start><jupyter_text>Principal Component Analysis in Python<jupyter_code># Data manipulation
import numpy as np
import pandas as pd

# Technical indicators
import talib as ta

# Plotting the graphs
import matplotlib.pyplot as plt

# Importing the machine learning libraries
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans, DBSCAN
from sklearn.manifold import TSNE
from sklearn import preprocessing
from statsmodels.tsa.stattools import coint

# The data is stored in the directory 'data_modules'
path = "../data_modules/"

# Import the dataset
df = pd.read_csv(path + "pca.csv", index_col=0)
df.head()<jupyter_output><empty_output><jupyter_text>Since we will be working on daily returns, we write the code as follows:<jupyter_code>data_daily_returns = df.pct_change()
data_daily_returns.dropna(inplace=True)<jupyter_output><empty_output><jupyter_text>If we have to check the number of rows and columns of the array, we use the following code:<jupyter_code>data_daily_returns.shape<jupyter_output><empty_output><jupyter_text>Here, we understand that there are 20 columns corresponding to the number of companies we have selected and 501 is the data points we have of each company. Moving ahead, we will now use the Principal Component Analysis code. Since we are trying to reduce the variables, let’s keep the number of Principal components as 18.<jupyter_code>N_PRIN_COMPONENTS = 18
pca = PCA(n_components=N_PRIN_COMPONENTS)
pca.fit(data_daily_returns)<jupyter_output><empty_output><jupyter_text>You can check the number of rows and columns in the array with the “shape” command.<jupyter_code>pre_process = pca.components_.T 
pre_process.shape<jupyter_output><empty_output><jupyter_text>You can see that we have gone from 501 to 20. Now, we have to use this for trading. First we will scale the data.<jupyter_code># Using transpose on the dataset
X = pca.components_.T
X.shape

# Scaling the dataset
X = preprocessing.StandardScaler().fit_transform(X)
print(X.shape)
# Using k-means algorithm on dataset
clf = KMeans(n_clusters=4, init='k-means++', max_iter=30,
             n_init=10, random_state=7)
print(clf)

# Using the fit function
clf.fit(X)
labels = clf.labels_
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)
print("\nClusters: %d" % n_clusters_)

clustered = clf.labels_
clustered<jupyter_output>KMeans(max_iter=30, n_clusters=4, random_state=7)

Clusters: 4<jupyter_text>To visualise it, we would use the t-SNE tool which is used to visualise high dimensional data into a 2D data.The exact code is as follows:<jupyter_code>clustered_series = pd.Series(index=data_daily_returns.columns, 
                             data=clustered.flatten())
clustered_series_all = pd.Series(
                            index=data_daily_returns.columns,
                            data=clustered.flatten())
clustered_series = clustered_series[clustered_series != -1]

X_tsne = TSNE(learning_rate=1000, perplexity=25, 
              random_state=1337).fit_transform(X)


plt.figure(1, facecolor='white')
plt.clf()
plt.axis('off')

plt.scatter(
    X_tsne[(labels!=-1), 0],
    X_tsne[(labels!=-1), 1],
    s=100,
    alpha=0.85,
    c=labels[labels!=-1]
)

plt.scatter(
    X_tsne[(clustered_series_all==-1).values, 0],
    X_tsne[(clustered_series_all==-1).values, 1],
    s=100,
    alpha=0.05
)

plt.title('T-SNE of all Stocks with KMeans Clusters Noted');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09446064139941691, 'fraction_numerical': 0.018367346938775512, 'mean_word_length': 4.8649572649572645, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '434417', 'license': 'CC0-1.0', 'n_tokens_mistral': 1222, 'n_tokens_neox': 1149, 'n_words': 322, 'path': 'Python Codes/Chapter 19 - Principal Component Analysis.ipynb', 'repo_name': 'quantra-go-algo/ml-trading-ebook', 'size': '3430'}