<jupyter_start><jupyter_text>Demo for data samplingIn this notebook we present sampling functionalities of the `preprocess` module. SamplingA synthetic data set is created and we perform train and test data sampling using available functions from the `DataSampler` class:- [**Section 1**](select_fixed_number): Fixed number of samples from each cluster (`DataSampler.number`)- [**Section 2**](select_fixed_percentage): Percentage of samples from each cluster (`DataSampler.percentage`)- [**Section 3**](select_manually): Manual sampling (`DataSampler.manual`)- [**Section 4**](select_at_random): Random sampling (`DataSampler.random`)For each function we present visually how test data can be sampled using the available `test_selection_option` parameter.[**Section 5**](maintain_fixed_test_data): We present a use-case for maintaining fixed test data samples using the `idx_test` parameter.[**Section 6**](chaining_sampling_functions): We show how two sampling functions can be combined where train samples from one sampling become fixed test data for another sampling.*** **Should plots be saved?**<jupyter_code>save_plots = False<jupyter_output><empty_output><jupyter_text>***<jupyter_code>from PCAfold import DataSampler
from PCAfold import preprocess
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import numpy as np

save_filename = None
color_map = ListedColormap(['#0e7da7', '#ceca70', '#b45050', '#2d2d54'])
first_cluster = False<jupyter_output><empty_output><jupyter_text>Generate data set that has four distinct clusters with imbalanced number of samples in each cluster:<jupyter_code>N_1, N_2, N_3, N_4 = 100, 250, 400, 500
n_observations = N_1 + N_2 + N_3 + N_4
mean_k1, mean_k2, mean_k3, mean_k4 = [-3, 3], [3, 3], [-3, -3], [3, -3] 
covariance = [[1, 0.2], [0.2, 1]]
x_k1, y_k1 = np.random.multivariate_normal(mean_k1, covariance, N_1).T
x_k2, y_k2 = np.random.multivariate_normal(mean_k2, covariance, N_2).T
x_k3, y_k3 = np.random.multivariate_normal(mean_k3, covariance, N_3).T
x_k4, y_k4 = np.random.multivariate_normal(mean_k4, covariance, N_4).T
x = np.vstack((x_k1[:,np.newaxis], x_k2[:,np.newaxis], x_k3[:,np.newaxis], x_k4[:,np.newaxis]))
y = np.vstack((y_k1[:,np.newaxis], y_k2[:,np.newaxis], y_k3[:,np.newaxis], y_k4[:,np.newaxis]))
idx = np.vstack((np.zeros((N_1, 1)), np.ones((N_2, 1)), 2*np.ones((N_3, 1)), 3*np.ones((N_4, 1)))).astype(int).ravel()
populations = preprocess.get_populations(idx)<jupyter_output><empty_output><jupyter_text>Plot the original data set:<jupyter_code>if save_plots: save_filename = '../images/tutorial-train-test-select-original-data-set'
preprocess.plot_2d_clustering(x, y, idx, x_label='$x$', y_label='$y$', color_map=color_map, first_cluster_index_zero=first_cluster, figure_size=(10,10), title='Original data set', save_filename=save_filename)<jupyter_output><empty_output><jupyter_text>Initialize the `DataSampler` class object:<jupyter_code>sample = DataSampler(idx, idx_test=[], random_seed=None, verbose=True)<jupyter_output><empty_output><jupyter_text>*** Select fixed number[**Go up**](header)Using `test_selection_option=1`:<jupyter_code>(idx_train, idx_test) = sample.number(15, test_selection_option=1)
if save_plots: save_filename = '../images/tutorial-train-test-select-fixed-number-1'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 46 train samples out of 100 observations (46.0%).
Cluster 1: taking 46 train samples out of 250 observations (18.4%).
Cluster 2: taking 46 train samples out of 400 observations (11.5%).
Cluster 3: taking 46 train samples out of 500 observations (9.2%).

Cluster 0: taking 54 test samples out of 54 remaining observations (100.0%).
Cluster 1: taking 204 test samples out of 204 remaining observations (100.0%).
Cluster 2: taking 354 test samples out of 354 remaining observations (100.0%).
Cluster 3: taking 454 test samples out of 454 remaining observations (100.0%).

Selected 184 train samples (14.7%) and 1066 test samples (85.3%).<jupyter_text>Using `test_selection_option=2`:<jupyter_code>(idx_train, idx_test) = sample.number(15, test_selection_option=2)
if save_plots: save_filename = '../images/tutorial-train-test-select-fixed-number-2'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 46 train samples out of 100 observations (46.0%).
Cluster 1: taking 46 train samples out of 250 observations (18.4%).
Cluster 2: taking 46 train samples out of 400 observations (11.5%).
Cluster 3: taking 46 train samples out of 500 observations (9.2%).

Cluster 0: taking 54 test samples out of 54 remaining observations (100.0%).
Cluster 1: taking 54 test samples out of 204 remaining observations (26.5%).
Cluster 2: taking 54 test samples out of 354 remaining observations (15.3%).
Cluster 3: taking 54 test samples out of 454 remaining observations (11.9%).

Selected 184 train samples (14.7%) and 216 test samples (17.3%).<jupyter_text>*** Select fixed percentage[**Go up**](header)Using `test_selection_option=1`:<jupyter_code>(idx_train, idx_test) = sample.percentage(10, test_selection_option=1)
if save_plots: save_filename = '../images/tutorial-train-test-select-fixed-percentage-1'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 10 train samples out of 100 observations (10.0%).
Cluster 1: taking 25 train samples out of 250 observations (10.0%).
Cluster 2: taking 40 train samples out of 400 observations (10.0%).
Cluster 3: taking 50 train samples out of 500 observations (10.0%).

Cluster 0: taking 90 test samples out of 90 remaining observations (100.0%).
Cluster 1: taking 225 test samples out of 225 remaining observations (100.0%).
Cluster 2: taking 360 test samples out of 360 remaining observations (100.0%).
Cluster 3: taking 450 test samples out of 450 remaining observations (100.0%).

Selected 125 train samples (10.0%) and 1125 test samples (90.0%).<jupyter_text>Using `test_selection_option=2`:<jupyter_code>(idx_train, idx_test) = sample.percentage(10, test_selection_option=2)
if save_plots: save_filename = '../images/tutorial-train-test-select-fixed-percentage-2'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 10 train samples out of 100 observations (10.0%).
Cluster 1: taking 25 train samples out of 250 observations (10.0%).
Cluster 2: taking 40 train samples out of 400 observations (10.0%).
Cluster 3: taking 50 train samples out of 500 observations (10.0%).

Cluster 0: taking 10 test samples out of 90 remaining observations (11.1%).
Cluster 1: taking 25 test samples out of 225 remaining observations (11.1%).
Cluster 2: taking 40 test samples out of 360 remaining observations (11.1%).
Cluster 3: taking 50 test samples out of 450 remaining observations (11.1%).

Selected 125 train samples (10.0%) and 125 test samples (10.0%).<jupyter_text>*** Select manually[**Go up**](header)Using `test_selection_option=1`:<jupyter_code>(idx_train, idx_test) = sample.manual({0:4, 1:5, 2:10, 3:2}, sampling_type='number', test_selection_option=1)
if save_plots: save_filename = '../images/tutorial-train-test-select-manually-1'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 4 train samples out of 100 observations (4.0%).
Cluster 1: taking 5 train samples out of 250 observations (2.0%).
Cluster 2: taking 10 train samples out of 400 observations (2.5%).
Cluster 3: taking 2 train samples out of 500 observations (0.4%).

Cluster 0: taking 96 test samples out of 96 remaining observations (100.0%).
Cluster 1: taking 245 test samples out of 245 remaining observations (100.0%).
Cluster 2: taking 390 test samples out of 390 remaining observations (100.0%).
Cluster 3: taking 498 test samples out of 498 remaining observations (100.0%).

Selected 21 train samples (1.7%) and 1229 test samples (98.3%).<jupyter_text>Using `test_selection_option=2`:<jupyter_code>(idx_train, idx_test) = sample.manual({0:50, 1:10, 2:10, 3:20}, sampling_type='percentage', test_selection_option=2)
if save_plots: save_filename = '../images/tutorial-train-test-select-manually-2'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 50 train samples out of 100 observations (50.0%).
Cluster 1: taking 25 train samples out of 250 observations (10.0%).
Cluster 2: taking 40 train samples out of 400 observations (10.0%).
Cluster 3: taking 100 train samples out of 500 observations (20.0%).

Cluster 0: taking 50 test samples out of 50 remaining observations (100.0%).
Cluster 1: taking 25 test samples out of 225 remaining observations (11.1%).
Cluster 2: taking 40 test samples out of 360 remaining observations (11.1%).
Cluster 3: taking 100 test samples out of 400 remaining observations (25.0%).

Selected 215 train samples (17.2%) and 215 test samples (17.2%).<jupyter_text>*** Select at random[**Go up**](header)Using `test_selection_option=1`:<jupyter_code>(idx_train, idx_test) = sample.random(10, test_selection_option=1)
if save_plots: save_filename = '../images/tutorial-train-test-select-random-doc-1'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 11 train samples out of 100 observations (11.0%).
Cluster 1: taking 25 train samples out of 250 observations (10.0%).
Cluster 2: taking 29 train samples out of 400 observations (7.2%).
Cluster 3: taking 60 train samples out of 500 observations (12.0%).

Cluster 0: taking 89 test samples out of 89 remaining observations (100.0%).
Cluster 1: taking 225 test samples out of 225 remaining observations (100.0%).
Cluster 2: taking 371 test samples out of 371 remaining observations (100.0%).
Cluster 3: taking 440 test samples out of 440 remaining observations (100.0%).

Selected 125 train samples (10.0%) and 1125 test samples (90.0%).<jupyter_text>Using `test_selection_option=2`:<jupyter_code>(idx_train, idx_test) = sample.random(10, test_selection_option=2)
if save_plots: save_filename = '../images/tutorial-train-test-select-random-doc-2'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 11 train samples out of 100 observations (11.0%).
Cluster 1: taking 30 train samples out of 250 observations (12.0%).
Cluster 2: taking 31 train samples out of 400 observations (7.8%).
Cluster 3: taking 53 train samples out of 500 observations (10.6%).

Cluster 0: taking 11 test samples out of 89 remaining observations (12.4%).
Cluster 1: taking 21 test samples out of 220 remaining observations (9.5%).
Cluster 2: taking 37 test samples out of 369 remaining observations (10.0%).
Cluster 3: taking 56 test samples out of 447 remaining observations (12.5%).

Selected 125 train samples (10.0%) and 125 test samples (10.0%).<jupyter_text>*** Maintain fixed test data[**Go up**](header) In the example below we further illustrate how the `idx_test` input parameter can be utilized when initializing `DataSampler` class.For that purpose we append the data set with additional samples. The additional samples simulate a real use-case where you have a very specific test set that you would like to maintain.We demonstrate this using each sampling function.<jupyter_code>Nt_1, Nt_2, Nt_3, Nt_4 = 20, 20, 20, 20
mean_k1, mean_k2, mean_k3, mean_k4 = [-7, 7], [7, 7], [-7, -7], [7, -7]
covariance = [[0.2, .1], [.1, 0.2]]
xt_k1, yt_k1 = np.random.multivariate_normal(mean_k1, covariance, Nt_1).T
xt_k2, yt_k2 = np.random.multivariate_normal(mean_k2, covariance, Nt_2).T
xt_k3, yt_k3 = np.random.multivariate_normal(mean_k3, covariance, Nt_3).T
xt_k4, yt_k4 = np.random.multivariate_normal(mean_k4, covariance, Nt_4).T
xt = np.vstack((xt_k1[:,np.newaxis], xt_k2[:,np.newaxis], xt_k3[:,np.newaxis], xt_k4[:,np.newaxis], x))
yt = np.vstack((yt_k1[:,np.newaxis], yt_k2[:,np.newaxis], yt_k3[:,np.newaxis], yt_k4[:,np.newaxis], y))
idx_appended = np.vstack((np.zeros((Nt_1, 1)), np.ones((Nt_2, 1)), 2*np.ones((Nt_3, 1)), 3*np.ones((Nt_4, 1)), np.zeros((N_1, 1)), np.ones((N_2, 1)), 2*np.ones((N_3, 1)), 3*np.ones((N_4, 1)))).astype(int).ravel()
idx_test = np.arange(0,Nt_1+Nt_2+Nt_3+Nt_4,1)
populations = preprocess.get_populations(idx_appended)
if save_plots: save_filename = '../images/tutorial-train-test-select-original-data-set-appended-doc'
preprocess.plot_2d_clustering(xt, yt, idx_appended, x_label="$x_t$", y_label="$y_t$", color_map=color_map, first_cluster_index_zero=first_cluster, figure_size=(10,10), title='Original data set', save_filename=save_filename)<jupyter_output><empty_output><jupyter_text>Initialize the `DataSampler` class object using `idx_test` as an input parameter:<jupyter_code>sample = DataSampler(idx_appended, idx_test=idx_test, random_seed=None, verbose=True)<jupyter_output>User defined test samples will be used. Parameter `test_selection_option` will be ignored.<jupyter_text>Select samples using each of the sampling functions (note that `test_selection_option` will be ignored in each case when `idx_test` parameter is used):<jupyter_code>(idx_train, idx_test) = sample.number(10, test_selection_option=2)
if save_plots: save_filename = '../images/tutorial-train-test-select-number-with-idx-test'
preprocess.plot_2d_train_test_samples(xt, yt, idx_appended, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)
(idx_train, idx_test) = sample.percentage(10, test_selection_option=2)
if save_plots==True: save_filename = '../images/tutorial-train-test-select-percentage-with-idx-test'
preprocess.plot_2d_train_test_samples(xt, yt, idx_appended, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)
(idx_train, idx_test) = sample.manual({0:10, 1:10, 2:10, 3:10}, sampling_type='percentage', test_selection_option=2)
if save_plots: save_filename = '../images/tutorial-train-test-select-manual-with-idx-test'
preprocess.plot_2d_train_test_samples(xt, yt, idx_appended, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)
(idx_train, idx_test) = sample.random(80, test_selection_option=2)
if save_plots: save_filename = '../images/tutorial-train-test-select-random-with-idx-test-doc'
preprocess.plot_2d_train_test_samples(xt, yt, idx_appended, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>Cluster 0: taking 91 train samples out of 120 observations (75.8%).
Cluster 1: taking 207 train samples out of 270 observations (76.7%).
Cluster 2: taking 341 train samples out of 420 observations (81.2%).
Cluster 3: taking 425 train samples out of 520 observations (81.7%).

Cluster 0: taking 20 test samples out of 29 remaining observations (69.0%).
Cluster 1: taking 20 test samples out of 63 remaining observations (31.7%).
Cluster 2: taking 20 test samples out of 79 remaining observations (25.3%).
Cluster 3: taking 20 test samples out of 95 remaining observations (21.1%).

Selected 1064 train samples (80.0%) and 80 test samples (6.0%).<jupyter_text>*** Chaining sampling functions[**Go up**](header)Finally, we will discuss an interesting use-case for chaining two sampling functions where train samples obtained from one sampling can become fixed test data for another sampling.Suppose that our target is to have a fixed test data set that will be composed of:- 10 samples from the first cluster- 20 samples from the second cluster- 10 samples from the third cluster- 50 samples from the fourth clusterand at the same time generate fixed number of samples from each cluster as train data.We can start with generating desired test samples using `DataSampler.manual` function. We can output train data as test data:<jupyter_code>sample = DataSampler(idx, random_seed=None, verbose=True)
(idx_test, _) = sample.manual({0:10, 1:20, 2:10, 3:50}, sampling_type='number', test_selection_option=1)<jupyter_output>Cluster 0: taking 10 train samples out of 100 observations (10.0%).
Cluster 1: taking 20 train samples out of 250 observations (8.0%).
Cluster 2: taking 10 train samples out of 400 observations (2.5%).
Cluster 3: taking 50 train samples out of 500 observations (10.0%).

Cluster 0: taking 90 test samples out of 90 remaining observations (100.0%).
Cluster 1: taking 230 test samples out of 230 remaining observations (100.0%).
Cluster 2: taking 390 test samples out of 390 remaining observations (100.0%).
Cluster 3: taking 450 test samples out of 450 remaining observations (100.0%).

Selected 90 train samples (7.2%) and 1160 test samples (92.8%).<jupyter_text>Now we feed the obtained test set as a fixed test set for the target sampling:<jupyter_code>sample.idx_test = idx_test
(idx_train, idx_test) = sample.number(19.5, test_selection_option=1)
if save_plots: save_filename = '../images/tutorial-train-test-select-chaining-functions'
preprocess.plot_2d_train_test_samples(x, y, idx, idx_train, idx_test, color_map=color_map, first_cluster_index_zero=first_cluster, save_filename=save_filename)<jupyter_output>User defined test samples will be used. Parameter `test_selection_option` will be ignored.

Cluster 0: taking 60 train samples out of 100 observations (60.0%).
Cluster 1: taking 60 train samples out of 250 observations (24.0%).
Cluster 2: taking 60 train samples out of 400 observations (15.0%).
Cluster 3: taking 60 train samples out of 500 observations (12.0%).

Cluster 0: taking 10 test samples out of 40 remaining observations (25.0%).
Cluster 1: taking 20 test samples out of 190 remaining observations (10.5%).
Cluster 2: taking 10 test samples out of 340 remaining observations (2.9%).
Cluster 3: taking 50 test samples out of 440 remaining observations (11.4%).

Selected 240 train samples (19.2%) and 90 test samples (7.2%).
{'fraction_non_alphanumeric': 0.11250471012542392, 'fraction_numerical': 0.06642622597836034, 'mean_word_length': 6.299803536345776, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '425191', 'license': 'MIT', 'n_tokens_mistral': 6829, 'n_tokens_neox': 5796, 'n_words': 1949, 'path': 'docs/tutorials/demo-sampling.ipynb', 'repo_name': 'ElsevierSoftwareX/SOFTX-D-20-00048', 'size': '18577'}