<jupyter_start><jupyter_text>Using Scikit Learn for Preprocessing Data Lets assume that data is the data we have. If we are going to use this data in our neural network model one of the first things that we are going to do is to scale the data.<jupyter_code>scaler_model = MinMaxScaler()<jupyter_output><empty_output><jupyter_text>Basically create a `MinMaxScalar` instance<jupyter_code>scaler_model.fit(data) # we are fitting the data to the MinMaxScaler model
scaler_model.transform(data) # transforming the data<jupyter_output><empty_output><jupyter_text><jupyter_code># Here as we can clearly see that the minimum value has become zero and the maximum value has become 1. 
# There are obviously other scalers found
scaler_model.fit_transform(data)<jupyter_output>C:\Users\Rajdeep\AppData\Local\conda\conda\envs\tfdeeplearning\lib\site-packages\sklearn\utils\validation.py:475: DataConversionWarning: Data with input dtype int32 was converted to float64 by MinMaxScaler.
  warnings.warn(msg, DataConversionWarning)<jupyter_text>The **ScalerModel** should only be fitted to your training data. The idea is that we need to fit to our training data and thentransform our training data and then transform our test data<jupyter_code>import pandas as pd
mydata = np.random.randint(0,101,(50,4))
data = pd.DataFrame(data=mydata, columns=['f1', 'f2', 'f3', 'label'])<jupyter_output><empty_output><jupyter_text>Here the first 3 columns are essentially columns of the 3 features and the label is a separate label<jupyter_code>data
# lets split this into a training and test dataset
X = data[['f1', 'f2', 'f3']]
y = data['label']
# now we have the features and the labels
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
# test_size is simply the percentage of the data that we want to go to the test set
# random_state is simply like the seed in a random number generator, the idea is to always shuffle before getting into the train
# and test sets
type(X_train)
X_train.shape
X_test.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07641196013289037, 'fraction_numerical': 0.013289036544850499, 'mean_word_length': 6.219178082191781, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '135111', 'license': 'MIT', 'n_tokens_mistral': 667, 'n_tokens_neox': 649, 'n_words': 244, 'path': 'ScikitLearnPreprocessing/ScikitLearnPreprocessing.ipynb', 'repo_name': 'Diagon-Alley/tensor-flow-exps', 'size': '2107'}