<jupyter_start><jupyter_text>Feature Scaling While many algorithms (such as SVM, K-nearest neighbors, and logistic regression) require features to be normalized, intuitively we can think of **Principle Component Analysis (PCA)** as being a prime example of when normalization is important. In PCA we are interested in the components that maximize the variance. **If one component (e.g. human height) varies less than another (e.g. weight) because of their respective scales (meters vs. kilos)**, PCA might determine that the direction of maximal variance more closely corresponds with the ‘weight’ axis, if those features are **not scaled**. As a change in height of one meter can be considered much more important than the change in weight of one kilogram, this is clearly **incorrect**.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn import preprocessing
from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn import metrics

%matplotlib inline<jupyter_output><empty_output><jupyter_text>We will use a small dataset that contains (Physics,Biology and Maths) marks of a classroom of students.<jupyter_code>df = pd.read_csv("https://archive.org/download/ml-fundamentals-data/machine-learning-fundamentals-data/grades.csv", index_col=0)<jupyter_output><empty_output><jupyter_text>Show the first 5 rows of data.<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>We can use boxplot to visualize the data<jupyter_code>df.boxplot()<jupyter_output><empty_output><jupyter_text>We could notice that the data is spread around the range of 1 to 100 We will use scaling functions from scikit learn to perform some preprocessing techniques to scale our data.**Min-Max normalization** involves scaling features to lie between a given minimum and maximum value, often between zero and one.<jupyter_code>scaler = preprocessing.MinMaxScaler()<jupyter_output><empty_output><jupyter_text>We will use Min-Max Scaling to scale all the columns of data.<jupyter_code>data_scaled = scaler.fit_transform(df)<jupyter_output><empty_output><jupyter_text>Transform the numpy array containing our scaled data into a pandas data frame.<jupyter_code>df_new = pd.DataFrame(data_scaled, index=df.index)<jupyter_output><empty_output><jupyter_text>Shows the first 5 rows of the scaled data<jupyter_code>df_new.head()<jupyter_output><empty_output><jupyter_text>As you can see, our values are scaled into a *range from 0 to 1*.<jupyter_code>df_new.boxplot()<jupyter_output><empty_output><jupyter_text>Another common scaling technique is the **Standard Scaler**.Standard Scaler Standardize features by removing the mean and scaling to unit varianceThe standard score of a sample x is calculated as:$$z = \frac{x - u}{s}$$where $u$ is the mean of the training samples or zero if `with_mean = False`, and $s$ is the standard deviation of the training samples or one if `with_std = False`.**Standardization** of a dataset is a common requirement for many machine learning estimators: they might behave badly if the individual features do not more or less look like standard normally distributed data (e.g. Gaussian with 0 mean and unit variance). Now use the standard scaler to scale the data<jupyter_code>scaler = preprocessing.StandardScaler()<jupyter_output><empty_output><jupyter_text>Apply the scaler on the dataset and use df.head to visual the data<jupyter_code>data_scaled = scaler.fit_transform(df)
df_new = pd.DataFrame(data_scaled, index=df.index)
df_new.head()<jupyter_output><empty_output><jupyter_text>By plotting boxplot we can see that the range of the data changed where their average mean will be 0<jupyter_code>df_new.boxplot()<jupyter_output><empty_output><jupyter_text>Example of improved performance using Standard Scaling  In this example, we will use a social network advertisment dataset which the result will be whether the targeted person purchased the items in the advertisement<jupyter_code>df = pd.read_csv("data/socialnetads.csv")<jupyter_output><empty_output><jupyter_text>Visualizing the first 5 example of the dataset<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Dropping column User ID which are not needed in the training<jupyter_code>df.drop(columns=['User ID'], axis=1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Use One Hot Encoding to split the Gender attribute into binary values<jupyter_code>df = pd.get_dummies(df, columns=['Gender'])
df.head()<jupyter_output><empty_output><jupyter_text>Assigning the attributes that are needed for training<jupyter_code>df.columns
y = df['Purchased']
df.drop(columns=['Purchased'], axis=1, inplace=True)
X = df<jupyter_output><empty_output><jupyter_text>Splitting the data into training set and test set<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=123)<jupyter_output><empty_output><jupyter_text>Training the model with Logistic Regression<jupyter_code>model = LogisticRegression()
model.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Get the prediction using the test set<jupyter_code>predict = model.predict(X_test)<jupyter_output><empty_output><jupyter_text>Calculating the accuracy score for the trained model<jupyter_code>accuracy = metrics.accuracy_score(predict, y_test)
print(accuracy)<jupyter_output>0.6416666666666667<jupyter_text>As from here we can see the the performance of the model is not that good, if we use feature scaling, the performance will be better. Scale the training set data with Standard Scaler We will only fit the training set only as we learn the means and standard deviation of the training set, and then:- Standardize the training set using the training set means and standard deviations.- Standardize the test set using the training set means and standard deviations.Anything that we want to learn, must be learned from the model's training data, this is because we want to keep the test set data to be "unknown" by the model. If we scale the training set and test set together, we are actually using the knowledge of the whole data which result in "leaking information"Besides that,  the test set data should be using the scaler from  the training set data . If we scale the training set data and test set data separately, the value of 1.0 in training set data and 1.0 in test set data would actually be different as they learned from each respective mean and standard deviation. In the end it would greatly impact the model performance.<jupyter_code>scaler = preprocessing.StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Now let us train the model again with the scaled data and look at the output of the accuracy<jupyter_code>model.fit(X_train, y_train)
predict = model.predict(X_test)
accuracy = metrics.accuracy_score(predict, y_test)
print(accuracy)<jupyter_output>0.8666666666666667
{'fraction_non_alphanumeric': 0.07162181403958422, 'fraction_numerical': 0.008116189662537377, 'mean_word_length': 6.865621500559911, 'pattern_counts': {'":': 0, '<': 91, '<?xml version=': 0, '>': 91, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '268237', 'license': 'MIT', 'n_tokens_mistral': 2109, 'n_tokens_neox': 2025, 'n_words': 773, 'path': 'solution/03-0_Feature Scaling.ipynb', 'repo_name': 'constantimi/Machine-Learning-Fundamentals', 'size': '7023'}