<jupyter_start><jupyter_text>Principal Component AnalysisGoal:  can we distill all columns of weather-stations, holidays and days of week down to just a few dimensions that would be good to drive through some of the scikit-learn algorithms?   Run time:  5 seconds<jupyter_code>from sklearn.decomposition import PCA
import pylab as pl
from itertools import cycle
import pandas as pd
import numpy as np
# read target df from local 
target_df = pd.read_csv("dataframes/target_df.csv.gz", compression="infer", parse_dates = ["opday"])

# read target_df directly from github
# target_df = pd.read_csv("https://github.com/k5dru/CPSC_5383/blob/main/dataframes/target_df.csv.gz?raw=true", compression="gzip", parse_dates = ["opday"])
# what if I used EVERYTHING in my original DF? 
target_df = target_df.dropna()
y = target_df['sum_spp_load']

# drop the target, "opday" which is a sequential date, and the SUM_CDD and SUM_HDD columns (if they exist)
try: 
    Xdf = target_df.drop(['sum_spp_load', 'opday', 'SUM_CDD', 'SUM_HDD'],axis=1)
except: 
    Xdf = target_df.drop(['sum_spp_load', 'opday'],axis=1)
# from https://www.ritchieng.com/pandas-scikit-learn/ convert pandas to numpy
feature_cols = Xdf.columns.values
X = Xdf.loc[:, feature_cols]
X.shape
feature_cols<jupyter_output><empty_output><jupyter_text>The data set has 1535 samples (individual operating dates) in it. It has 481 dimensions; some are very similar to others due to locality of weather stations.<jupyter_code>pca = PCA(n_components=15, whiten=True).fit(X)
X_pca = pca.transform(X)<jupyter_output><empty_output><jupyter_text>What we have done is distill our 481D data set down to 4D.<jupyter_code>print(pca.components_)
X_pca.shape<jupyter_output><empty_output><jupyter_text>Let's see how much information we've managed to preserve:<jupyter_code>print(pca.explained_variance_ratio_)
print(sum(pca.explained_variance_ratio_))<jupyter_output>[0.87350987 0.02950356 0.02366626 0.01795975 0.00749497 0.00438432
 0.0039636  0.0032106  0.00279651 0.0020353  0.00183208 0.00147412
 0.00141667 0.00133899 0.00123744]
0.9758240467825636<jupyter_text>Eh, only 94.4% of the variation in the dataset is captured in 4 dimensions. In testing this code, it takes 50 dimensions to capture 99% of the variation. This leads to me to believe that an alternative approach may be needed; I'll look into a genetic algorithm for feature selection for the appropriate weather stations to predict demand in the utility footprint.<jupyter_code># important numpy arrary manipulation operations: 
# http://www.cs.cornell.edu/courses/cs4670/2016sp/lectures/lec06_numpy.pdf
X_pca[:,1]



%matplotlib inline
from pylab import *
#pl.figure()
pl.figure(figsize=(8, 6), dpi=120)
pl.scatter(X_pca[:,0], y,  c="b", label="pca1", s=1)
pl.scatter(X_pca[:,1], y,  c="y", label="pca2", s=1)
pl.scatter(X_pca[:,2], y,  c="g", label="pca3", s=1)
pl.scatter(X_pca[:,3], y,  c="c", label="pca4", s=1)
plt.xlabel("PCA vector values")  # custom x label using matplotlib
plt.ylabel("Daily Electric Demand, MWh")
pl.legend()
pl.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1159090909090909, 'fraction_numerical': 0.06688311688311688, 'mean_word_length': 5.923595505617977, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '654537', 'license': 'MIT', 'n_tokens_mistral': 1248, 'n_tokens_neox': 1095, 'n_words': 308, 'path': '04_0_PCA.ipynb', 'repo_name': 'k5dru/CPSC_5383', 'size': '3080'}