<jupyter_start><jupyter_text>Standard scaling featuresStandardization of features according to: x'=(x-µ)/ σ. Sets mean to 0 and unit variance to 1. Z-score Feature Scaling: https://developers.google.com/machine-learning/data-prep/transform/normalizationGreek symbols in ascii are produced like so:µ = Alt + 230 σ = Alt + 229 Φ = Alt + 232 https://www.keynotesupport.com/internet/special-characters-greek-letters-symbols.shtml<jupyter_code>#  train the scaler ONLY on the training set. Then use it to scale train/test/val
scaler = preprocessing.StandardScaler()
X_train = scaler.fit_transform(X_train) # trains the scaler using fit on X_train, then transforms X_train as well
X_test = scaler.transform(X_test) # no fit, transforms using data from fit() stored in the scaler
X_val = scaler.transform(X_val)<jupyter_output><empty_output><jupyter_text>Univariate Selection, SelectKBest[sklearn.feature_selection.SelectKBest](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.htmlsklearn.feature_selection.SelectKBest) will select features according to the k highest scores. [sklearn.feature_selection.r_regression](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.r_regression.htmlsklearn.feature_selection.r_regression) will compute Pearson's correlation coefficient for each feature and the target. This is a scoring function to be fed into a feature selection procedure, and does not stand on its own. Use with SelectKBest or similar method. The cross correlation between each regressor and the target is computed as ((X[:, i] - mean(X[:, i])) * (y - mean_y)) / (std(X[:, i]) * std(y)).<jupyter_code>print("SelectKBest, using an Pearson's correlation coefficient scoring function:")

test = SelectKBest(score_func=r_regression, k=4)
fit = test.fit(X_train, y_train)
features = fit.transform(X_train)

print("correlation coefficients:\n",fit.scores_)

print("best 4 features:\n",test.get_feature_names_out(feature_names))

# convert variables to PyTorch tensor
X_train = torch.tensor(X_train, dtype=torch.float32)
X_test = torch.tensor(X_test, dtype=torch.float32)
X_val = torch.tensor(X_val, dtype=torch.float32)
y_train = torch.tensor(y_train, dtype=torch.float32)
y_test = torch.tensor(y_test, dtype=torch.float32)
y_val = torch.tensor(y_val, dtype=torch.float32)
print(X_train.shape)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10504201680672269, 'fraction_numerical': 0.01050420168067227, 'mean_word_length': 6.463949843260188, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '348535', 'license': 'MIT', 'n_tokens_mistral': 773, 'n_tokens_neox': 742, 'n_words': 197, 'path': '3_dataprep_featureengineering.ipynb', 'repo_name': 'fieryWalrus1002/potato', 'size': '2380'}