<jupyter_start><jupyter_text>Data Analysis FINM September Launch Homework Solution 5  Imports<jupyter_code>import pandas as pd
import numpy as np
import statsmodels.api as sm
from sklearn.linear_model import LinearRegression
from sklearn.decomposition import PCA
from sklearn.cross_decomposition import PLSRegression
from numpy.linalg import svd
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
import warnings
warnings.filterwarnings("ignore")<jupyter_output><empty_output><jupyter_text>Data<jupyter_code>data = pd.read_excel("../data/single_name_return_data.xlsx", sheet_name="total returns").set_index("Date")
data.head()
equities = data.drop(columns=['SPY', 'SHV'])
equities.head()<jupyter_output><empty_output><jupyter_text>1 Principal Components  1.1 **Calculate the principal components of the return series.** Using linear algebra:<jupyter_code>clean = equities - equities.mean()
u, s, vh = svd(clean)
factors = clean @ vh.T
factors.columns = np.arange(1,23)
factors<jupyter_output><empty_output><jupyter_text>Using a package:<jupyter_code>pca = PCA(svd_solver='full')
pca.fit(equities.values)

pca_factors = pd.DataFrame(pca.transform(equities.values), 
                           columns=['Factor {}'.format(i+1) for i in range(pca.n_components_)], 
                           index=equities.index)
pca_factors<jupyter_output><empty_output><jupyter_text>1.2**Report the eigenvalues associated with these principal components. Report each eigenvalue as a percentage of the sum of all the eigenvalues. This is the total variation each PCA explains.** Using linear algebra:<jupyter_code>PCA_eigenvals = pd.DataFrame(index=factors.columns, columns=['Eigen Value', 'Percentage Explained'])
PCA_eigenvals['Eigen Value'] = s**2
PCA_eigenvals['Percentage Explained'] = s**2 / (s**2).sum()
PCA_eigenvals<jupyter_output><empty_output><jupyter_text>Using package (no eignvalues method):<jupyter_code>pkg_explained_var = pd.DataFrame(data = pca.explained_variance_ratio_, 
                                 index = factors.columns, 
                                 columns = ['Explained Variance'])

pkg_explained_var<jupyter_output><empty_output><jupyter_text>1.3**How many PCs are needed to explain 75% of the variation?**<jupyter_code>pkg_explained_var.cumsum()<jupyter_output><empty_output><jupyter_text>We need the first 5 PCs to explain at least 75% of the variation  1.4**Calculate the correlation between the first (largest eigenvalue) principal component with each of the 22 single-name equities. Which correlation is highest?**<jupyter_code>corr_4 = equities.copy()
corr_4['factor 1'] = factors[1]

corr_equities = corr_4.corr()['factor 1'].to_frame('Correlation to factor 1')
corr_equities.iloc[:len(equities.columns)]<jupyter_output><empty_output><jupyter_text>1.5**Calculate the correlation between the SPY and the first, second, third principal components.**<jupyter_code>fac_corr = factors[[1,2,3]]
fac_corr['SPY'] = data['SPY']

SPY_corr = fac_corr.corr()['SPY'].to_frame('Correlation to SPY').iloc[:3]

SPY_corr<jupyter_output><empty_output><jupyter_text>2 PCR and PLS  2.1**Principal Component Regression (PCR) refers to using PCA for dimension reduction, and thenutilizing the principal components in a regression. Try this by regressing SPY on the first 3 PCscalculated in the previous section. Report the r-squared.**<jupyter_code>y_PCR = data['SPY']
X_PCR = factors[[1,2,3]]

model_PCR = LinearRegression().fit(X_PCR,y_PCR)
print('PCR R-squared: ' + str(round(model_PCR.score(X_PCR, y_PCR),3)))<jupyter_output>PCR R-squared: 0.922<jupyter_text>2.2**Calculate the Partial Least Squares estimation of SPY on the 22 single-name equities. Model it for 3 factors. Report the r-squared.**<jupyter_code>X_PLS = equities
y_PLS = data['SPY']

model_PLS = PLSRegression(n_components=3).fit(X_PLS, y_PLS)

print('PLS R-squared: ' + str(round(model_PLS.score(X_PLS, y_PLS),3)))<jupyter_output>PLS R-squared: 0.961
{'fraction_non_alphanumeric': 0.11458333333333333, 'fraction_numerical': 0.01524390243902439, 'mean_word_length': 6.042933810375671, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '6050', 'license': 'MIT', 'n_tokens_mistral': 1416, 'n_tokens_neox': 1319, 'n_words': 309, 'path': 'homework/Homework_5.ipynb', 'repo_name': 'standingtree/finm-analytics-2021', 'size': '3936'}