<jupyter_start><jupyter_text>Preprocessing TechniquesIn the examples below, four preprocessing techniques will be demonstrated:- Binarization- Mean removal- Scaling- NormalizationThe matrix below was used as the sample set:<jupyter_code>import numpy as np
from sklearn import preprocessing as pp

m = np.array([[ 5.1, -2.9,  3.3],
              [-1.2,  7.8, -6.1],
              [ 3.9,  0.4,  2.1],
              [ 7.3, -9.9, -4.5]])<jupyter_output><empty_output><jupyter_text>Binarization`preprocessing.Binarizer()``Binarizer()` method converts numerical values into booleans. Below, the `threshold` parameter is `2.1`. All values below 2.2 become `0`.<jupyter_code>b = pp.Binarizer().fit(m)
b2 = pp.Binarizer(threshold = 2.1).transform(m)

print("\nBinarizing - fit:\n", b)
print("\nBinarizing - threshold, transform:\n", b2)<jupyter_output>Binarizing - fit:
 Binarizer(copy=True, threshold=0.0)

Binarizing - threshold, transform:
 [[ 1.  0.  1.]
 [ 0.  1.  0.]
 [ 1.  0.  0.]
 [ 1.  0.  0.]]<jupyter_text>First, in order to illustrate the difference, the `mean()` and `std()` deviation are calculated below:<jupyter_code>print("\nBefore scaling:")
print("Mean:", m.mean(axis = 0))
print("Standard:", m.std(axis = 0))<jupyter_output>Before scaling:
Mean: [ 3.775 -1.15  -1.3  ]
Standard: [ 3.12039661  6.36651396  4.0620192 ]<jupyter_text>Mean RemovalIn order to center each feature on zero, the mean value is removed.`preprocessing.scale()`<jupyter_code>s = pp.scale(m)

print("\nAfter deviation:")
print("Mean:", s.mean(axis = 0))
print("Standard:", s.std(axis = 0))<jupyter_output>After deviation:
Mean: [  1.11022302e-16   0.00000000e+00   2.77555756e-17]
Standard: [ 1.  1.  1.]<jupyter_text>Minimum/Maximum ScalingScaling maintains uniformity.`preprocessing.MinMaxScaler()`<jupyter_code>smm = pp.MinMaxScaler(feature_range=(0, 1))
data_scaled_minmax = smm.fit_transform(m)

print("\nMin/max scaled:\n", data_scaled_minmax)<jupyter_output>Min/max scaled:
 [[ 0.74117647  0.39548023  1.        ]
 [ 0.          1.          0.        ]
 [ 0.6         0.5819209   0.87234043]
 [ 1.          0.          0.17021277]]<jupyter_text>NormalizationNormalization conforms values to a scale. For example, a row of values can be represented by decimal numbers which all add up to `1`, as with _L1 normalization_ or _least absolute deviations_. _L2 normalization_ is similar, but the sum of squared values is `1`. _L1 normalization_ is more effective, being resistant to outliers. But if outliers become important, then _L2_ might be preferred.`preprocessing.normalize()`<jupyter_code>nl1 = pp.normalize(m, norm='l1')
nl2 = pp.normalize(m, norm='l2')

print("\nL1 normalized:\n", nl1)
print("\nL2 normalized:\n", nl2)<jupyter_output>L1 normalized:
 [[ 0.45132743 -0.25663717  0.2920354 ]
 [-0.0794702   0.51655629 -0.40397351]
 [ 0.609375    0.0625      0.328125  ]
 [ 0.33640553 -0.4562212  -0.20737327]]

L2 normalized:
 [[ 0.75765788 -0.43082507  0.49024922]
 [-0.12030718  0.78199664 -0.61156148]
 [ 0.87690281  0.08993875  0.47217844]
 [ 0.55734935 -0.75585734 -0.34357152]]
{'fraction_non_alphanumeric': 0.14378659726740403, 'fraction_numerical': 0.1291476903057905, 'mean_word_length': 3.84251968503937, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '598599', 'license': 'MIT', 'n_tokens_mistral': 1377, 'n_tokens_neox': 1150, 'n_words': 279, 'path': 'python-note/python_notebooks-master/modules/data_preprocessor.ipynb', 'repo_name': 'ii6uu99/ipynb', 'size': '3074'}