<jupyter_start><jupyter_text>`PCAfold`: Python software to generate, analyze and improve PCA-derived low-dimensional manifolds Authors: **<NAME>**, **<NAME>**, **<NAME>**, **<NAME>***Université Libre de Bruxelles*, *The University of Utah*, 2020This Jupyter notebook contains code presented within the original software publication.***  Sample code snippetsCode presented in section *2.2 Software functionalities and sample code snippets*.  Data sampling<jupyter_code>from PCAfold import DataSampler
import numpy as np

# Generate a dummy data set:
X = np.random.rand(1000,3)

# Generate a dummy vector of cluster classifications:
idx = np.zeros((1000,))
idx[500:800] = 1

# Instantiate DataSampler class object:
selection = DataSampler(idx, idx_test=[], random_seed=100, verbose=True)

# (1) Select equal number of samples from each cluster:
(idx_train, idx_test) = selection.number(20, test_selection_option=1)

# (2) Select equal percentage of samples from each cluster:
(idx_train, idx_test) = selection.percentage(20, test_selection_option=1)

# (3) Select samples manually from each cluster:
(idx_train, idx_test) = selection.manual({0:200, 1:100},
sampling_type='number', test_selection_option=1)

# (4) Select samples at random from each cluster:
(idx_train, idx_test) = selection.random(20, test_selection_option=1)

# Partition the original observations into train and test data:
X_train = X[idx_train,:]
X_test = X[idx_test,:]<jupyter_output>Cluster 0: taking 100 train samples out of 700 observations (14.3%).
Cluster 1: taking 100 train samples out of 300 observations (33.3%).

Cluster 0: taking 600 test samples out of 600 remaining observations (100.0%).
Cluster 1: taking 200 test samples out of 200 remaining observations (100.0%).

Selected 200 train samples (20.0%) and 800 test samples (80.0%).

Cluster 0: taking 140 train samples out of 700 observations (20.0%).
Cluster 1: taking 60 train samples out of 300 observations (20.0%).

Cluster 0: taking 560 test samples out of 560 remaining observations (100.0%).
Cluster 1: taking 240 test samples out of 240 remaining observations (100.0%).

Selected 200 train samples (20.0%) and 800 test samples (80.0%).

Cluster 0: taking 200 train samples out of 700 observations (28.6%).
Cluster 1: taking 100 train samples out of 300 observations (33.3%).

Cluster 0: taking 500 test samples out of 500 remaining observations (100.0%).
Cluster 1: taking 200 test samples out of 200 remaining[...]<jupyter_text>Principal Component Analysis (PCA)<jupyter_code>from PCAfold import PCA
import numpy as np

# Generate a dummy data set:
X = np.random.rand(100,10)

# Instantiate PCA class object:
pca_X = PCA(X, scaling='auto', n_components=2)

# Eigenvectors:
eigenvectors = pca_X.A

# Eigenvalues:
eigenvalues = pca_X.L

# Principal Components:
principal_components = pca_X.transform(X)

# Reconstruct the data set from the first two Principal Components:
X_rec = pca_X.reconstruct(principal_components)<jupyter_output><empty_output><jupyter_text>*** Illustrative exampleCode presented in section *3. Illustrative example*.<jupyter_code>import numpy as np

# Original variables:
X = np.genfromtxt('docs/tutorials/data-state-space.csv', delimiter=',')

# List of names of the original variables:
variables_names = ['$T$', '$H_2$', '$O_2$', '$O$', '$OH$', '$H_2O$', '$H$', '$HO_2$', '$CO$', '$CO_2$', '$HCO$']

# Corresponding source terms of the original variables:
S_X = np.genfromtxt('docs/tutorials/data-state-space-sources.csv', delimiter=',')
from PCAfold import preprocess
from PCAfold import reduction
from PCAfold import analysis

# Perform PCA on the data set:
pca_X = reduction.PCA(X, scaling='auto', n_components=2)

# Transform original variables to the PC space:
Z = pca_X.transform(X, nocenter=False)

# Transform sources of the original variables to the PC space:
S_Z = pca_X.transform(S_X, nocenter=True)

# Cluster the data set:
idx = preprocess.zero_neighborhood_bins(S_Z[:,0], k=4, zero_offset_percentage=2, split_at_zero=True, verbose=True)

# Compute populations of each cluster:
populations = preprocess.get_populations(idx)
# Instantiate DataSampler class object:
sample = preprocess.DataSampler(idx, idx_test=[], random_seed=100, verbose=True)

# Select 2400 samples from each cluster:
(idx_manual, _) = sample.manual({0:2400, 1:2400, 2:2400, 3:2400}, sampling_type='number', test_selection_option=1)
# Perform PCA on a sampled data set according to biasing option 2:
(eigenvalues, eigenvectors, Z_r, S_Z_r, C, D, C_r, D_r) = reduction.pca_on_sampled_data_set(X, idx_manual, scaling='auto', n_components=2, biasing_option=2, X_source=S_X)
# Plot the weights change on the first eigenvector:
plt = reduction.analyze_eigenvector_weights_change(np.vstack((pca_X.A[:,0], eigenvectors[:,0])).T, variables_names, legend_label=['$\mathbf{X}$', '$\mathbf{X_r}$'], save_filename=None)

# Plot the initial two-dimensional manifold:
plt = reduction.plot_2d_manifold(Z[:,0], Z[:,1], color_variable=X[:,6], x_label='$Z_{1}$', y_label='$Z_{2}$', colorbar_label='$Y_H$ [-]', save_filename=None)

# Plot the biased two-dimensional manifold:
plt = reduction.plot_2d_manifold(Z_r[:,0], Z_r[:,1], color_variable=X[:,6], x_label='$Z_{r, 1}$', y_label='$Z_{r, 2}$', colorbar_label='$Y_H$ [-]', save_filename=None)
# Bandwidth values:
bandwidth_values = np.logspace(-3.5, 0.5, 25)

# Create dependent variables matrices:
depvars_initial = np.hstack((S_Z, X[:,[6]]))
depvars_biased = np.hstack((S_Z_r, X[:,[6]]))

# Create names for dependent variables:
depvar_names_initial = ['$S_{Z_1}$', '$S_{Z_2}$', '$Y_H$']
depvar_names_biased = ['$S_{Z_{r, 1}}}$', '$S_{Z_{r, 2}}$', '$Y_H$']
# Compute normalized variance quantities on the initial manifold:
variance_data_initial = analysis.compute_normalized_variance(Z, depvars_initial, depvar_names=depvar_names_initial, bandwidth_values=bandwidth_values)
# Compute normalized variance quantities on the biased manifold:
variance_data_biased = analysis.compute_normalized_variance(Z_r, depvars_biased, depvar_names=depvar_names_biased, bandwidth_values=bandwidth_values)
# Plot the comparison of normalized variance quantities:
plt = analysis.plot_normalized_variance_comparison((variance_data_initial, variance_data_biased), ([], []), ('Greys', 'Reds'), save_filename=None)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12878667724028547, 'fraction_numerical': 0.04203013481363997, 'mean_word_length': 6.662211421628189, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '269030', 'license': 'MIT', 'n_tokens_mistral': 2278, 'n_tokens_neox': 2023, 'n_words': 617, 'path': 'SoftwareX-PCAfold-example.ipynb', 'repo_name': 'ElsevierSoftwareX/SOFTX-D-20-00048', 'size': '6350'}