<jupyter_start><jupyter_text>Preprocessing Table of Contents1&nbsp;&nbsp;Data segmentation2&nbsp;&nbsp;Spliting training and test data3&nbsp;&nbsp;Normalization3.1&nbsp;&nbsp;Z-score normalization3.2&nbsp;&nbsp;Uniform normalization (L1)3.3&nbsp;&nbsp;Min-max normalization4&nbsp;&nbsp;Reference  Data segmentationNext, we show how to segment the data stream. We take 0.1 second as an example of time window, therefore, each segment contains 16=0.6 $\times$ 160 time steps as the sampling rate is 160 Hz. We set the overlapping rate as 50%, indicating there are 50% overlap between two adjacent segments.<jupyter_code>import numpy as np
import myimporter
from BCI_functions import *  # BCI_functions.ipynb contains some functions we might use multiple times in this tutorial

dataset_1=np.load('1.npy')  # load data

n_class = 2  # 0~9 classes ('10:rest' is not considered)
no_feature = 14  # the number of the features
segment_length = 16  # selected time window; 16=160*0.1

# segment data, check more details about the 'extract' function in BCI_functions.ipynb
data_seg = extract(dataset_1, n_classes=n_class, n_fea=no_feature, time_window=segment_length, moving=(segment_length/2))  # 50% overlapping

print('After segmentation, the shape of the data:', data_seg.shape)<jupyter_output>After segmentation, the shape of the data: (64, 225)<jupyter_text>Spliting training and test dataIn this part, we will show how to spliting the dataset into training data and test data. Just simply use the well-packed function from sklearn: https://scikit-learn.org/stable/.<jupyter_code># First, we separate the features and the label 

data_seg_feature = data_seg[:, :1024]
data_seg_label = data_seg[:, 1024:1025]

# split data
from sklearn.model_selection import train_test_split
train_feature, test_feature, train_label, test_label = train_test_split(data_seg_feature, data_seg_label, shuffle=True)
print('The shape of training feature and label:', train_feature.shape, train_label.shape,
     '\nThe shape of test feature and label:', test_feature.shape, test_label.shape)<jupyter_output>The shape of training feature and label: (48, 225) (48, 0) 
The shape of test feature and label: (16, 225) (16, 0)<jupyter_text>NormalizationIn this section, we introduce a number of commonly used normalization strategies including z-score, uniform, and min-max normalization.  The library we adopt in this part is scikit-learn, if you are new to this, please check more details from: https://scikit-learn.org/stable/modules/preprocessing.html* **Why we need normalization?**   It's important for some machine learning methods that all features are on the same scale (e.g., faster convergence in learning, more uniform influence for all weights), which may enable the model to achieve better results. * **Time-wise normalization vs. channel-wise normalization**   The time-wise normalization is to rescale data along the time-wise while the channel-wise is to rescale along the channel-wise to a consistent distribution. Normally, for EEG signals, we only focus on the time-wise normalization because of its high temporal resolution.       Take our data X of shape (n_samples, n_channels) as an example, time-wise normalization means scaling along the sample axis.    Z-score normalizationZ-score normalization (Standardization) gives you an idea of how far from the mean a data point is.   The equation for z-score normalization:$$x_i^{'}=\frac{x_{i}-\bar{x_{i}}}{\sigma }$$<jupyter_code>from sklearn.preprocessing import StandardScaler 
scaler1 = StandardScaler().fit(train_feature)
train_fea_norm1 = scaler1.transform(train_feature) # normalize the training data
test_fea_norm1 = scaler1.transform(test_feature) # normalize the test data

print('After normalization, the shape of training feature:', train_fea_norm1.shape,
      '\nAfter normalization, the shape of test feature:', test_fea_norm1.shape)<jupyter_output>After normalization, the shape of training feature: (48, 225) 
After normalization, the shape of test feature: (16, 225)<jupyter_text>Uniform normalization (L1) For the uniform normalization, it process the dataset in a way that the sum of the absolute value of each column (in our case) to be one.   The equation for uniform normalization:$$x_i^{'}=\frac{x_{i}}{\sum x_{j}}$$* Note, in _sklearn.preprocessing.Normalizer_， the default normalization is L2, so remember to set the normalizer to L1 normalization (norm='l1') when you want to use uniform normalization.* Mind the axis in this one. The normalizer seems to be defaultly scale on the channel-wise axis, in this case, we have to transpose the data in order to get the correct normalization.<jupyter_code>from sklearn.preprocessing import Normalizer
scaler2 = Normalizer(norm='l1').fit(train_feature.T)  
train_fea_norm2 = scaler2.transform(train_feature.T) 
test_fea_norm2 = scaler2.transform(test_feature.T) 

train_fea_norm2 = train_fea_norm2.T
test_fea_norm2 = test_fea_norm2.T

print('After normalization, the shape of training feature:', train_fea_norm2.shape,
      '\nAfter normalization, the shape of test feature:', test_fea_norm2.shape)<jupyter_output><empty_output><jupyter_text>Min-max normalization Min-max normalization is one of the most common ways to normalize data. For every feature, the minimum value of that feature gets transformed into a 0, the maximum value gets transformed into a 1, and every other value gets transformed into a decimal between 0 and 1.[1](refer-anchor-1)   The equation for min-max normalization:  $$x_i^{'}=\frac{x_{i}-x_{min}}{x_{max}-x_{min}}$$<jupyter_code>from sklearn.preprocessing import MinMaxScaler
scaler3 = MinMaxScaler().fit(train_feature)
train_fea_norm3 = scaler3.transform(train_feature)
test_fea_norm3 = scaler3.transform(test_feature)

print('After normalization, the shape of training feature:', train_fea_norm3.shape,
      '\nAfter normalization, the shape of test feature:', test_fea_norm3.shape)<jupyter_output>After normalization, the shape of training feature: (48, 225) 
After normalization, the shape of test feature: (16, 225)
{'fraction_non_alphanumeric': 0.083993399339934, 'fraction_numerical': 0.02277227722772277, 'mean_word_length': 5.8485875706214685, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '114123', 'license': 'MIT', 'n_tokens_mistral': 1849, 'n_tokens_neox': 1716, 'n_words': 672, 'path': 'tutorial/2-Preprocessing.ipynb', 'repo_name': 'rhkddlf/bci_project', 'size': '6060'}