<jupyter_start><jupyter_text>The goal of this notebook is to understand the differences between Scaling, Normalizing and Standardizing.  The visualizations only help us notice the problems visually.  Reference for inspiration :https://towardsdatascience.com/scale-standardize-or-normalize-with-scikit-learn-6ccc7d176a02  TODO :: Why are these three transformations important ??<jupyter_code>import numpy as np 
import pandas as pd 
from sklearn import preprocessing

import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.DataFrame({
    'beta' : np.random.beta(5, 1, 1000) * 60,
    'exponential' : np.random.exponential(10, 1000),
    'normal' : np.random.normal(10, 2, 1000),
    'big_normal' : np.random.normal(1000000, 10000, 1000)  
})

first = np.random.normal(20, 3, 500)
second = np.random.normal(-20, 3, 500)
bimodal = np.concatenate([first, second])

df['bimodal'] = bimodal 



col_names = df.columns
df.describe()
fig, (ax1, ax2) = plt.subplots(ncols=2, figsize = (20, 8))
ax1.set_title('Original Distributions without the big normal')
ax2.set_title('See how it gets all warped due to big_normal')

# plot on the left
sns.kdeplot(df['beta'], ax=ax1)
sns.kdeplot(df['bimodal'], ax=ax1)
sns.kdeplot(df['normal'], ax=ax1)
sns.kdeplot(df['exponential'], ax=ax1)


# plot on the right
sns.kdeplot(df['beta'], ax=ax2)
sns.kdeplot(df['bimodal'], ax=ax2)
sns.kdeplot(df['normal'], ax=ax2)
sns.kdeplot(df['big_normal'], ax=ax2)
sns.kdeplot(df['exponential'], ax=ax2)<jupyter_output><empty_output><jupyter_text>Things to consider while choosing the required transformation* Presence and logic to handle marginal outliers* Whether the different feature scales should be balanced? i.e. two different features will be scaled incomparably* Is it required by the analysis/ algorithm youre running   1. Scaling -- changing range* Bringing the data within a certain range. often [0-1] * Data's distribution remains unchanged * outlier still exert influence post scaling **MinMaxScaler*** sensitive to outliers , outliers still exert influence after scaling * features are not normally distributed * least disruptive when it comes to changing the shape of the data<jupyter_code>min_maxScaler = preprocessing.MinMaxScaler()
minmax = min_maxScaler.fit_transform(df)
minmax # returns a np array
minmax_df = pd.DataFrame(minmax, columns=[name+"_minMaxed" for name in col_names])
# plot on the left
fig, (ax1, ax3) = plt.subplots(ncols=2, figsize = (20, 8))
ax1.set_title('Original Distributions')
ax3.set_title('Distributions made comparable by MinMaxScaler')


sns.kdeplot(df['beta'], ax=ax1)
sns.kdeplot(df['bimodal'], ax=ax1)
sns.kdeplot(df['normal'], ax=ax1)
sns.kdeplot(df['big_normal'], ax=ax1)
sns.kdeplot(df['exponential'], ax=ax1)

# plot on the right
sns.kdeplot(minmax_df['beta_minMaxed'], ax=ax3)
sns.kdeplot(minmax_df['bimodal_minMaxed'], ax=ax3)
sns.kdeplot(minmax_df['normal_minMaxed'], ax=ax3)
sns.kdeplot(minmax_df['big_normal_minMaxed'], ax=ax3)
sns.kdeplot(minmax_df['exponential_minMaxed'], ax=ax3)<jupyter_output><empty_output><jupyter_text>2. Standardization -- changing the shape of the data (scaling inherent)* Bringing the distribution closer to normal -- changes the shape and distribution of the data * Reduces the influence of outliers* Standard deviation of transformed distribution should be closer to 1 **StandardScaler**TODO features<jupyter_code>std_scaler = preprocessing.StandardScaler()
stdscaled = std_scaler.fit_transform(df)
stdscaled_df = pd.DataFrame(stdscaled, columns=[name+"_stdscaled" for name in col_names])

stdscaled_df.describe()<jupyter_output><empty_output><jupyter_text>Notice how the standard deviation is now close to 1<jupyter_code>fig, (ax1, ax4) = plt.subplots(ncols=2, figsize = (20, 8))
ax1.set_title('Original Distributions')
ax3.set_title('Distributions made comparable by Standard Scaler')


sns.kdeplot(df['beta'], ax=ax1)
sns.kdeplot(df['bimodal'], ax=ax1)
sns.kdeplot(df['normal'], ax=ax1)
sns.kdeplot(df['big_normal'], ax=ax1)
sns.kdeplot(df['exponential'], ax=ax1)

# plot on the right
sns.kdeplot(stdscaled_df['beta_stdscaled'], ax=ax4)
sns.kdeplot(stdscaled_df['bimodal_stdscaled'], ax=ax4)
sns.kdeplot(stdscaled_df['normal_stdscaled'], ax=ax4)
sns.kdeplot(stdscaled_df['big_normal_stdscaled'], ax=ax4)
sns.kdeplot(stdscaled_df['exponential_stdscaled'], ax=ax4)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12611377655928718, 'fraction_numerical': 0.026273703449851495, 'mean_word_length': 6.8882882882882885, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '93980', 'license': 'MIT', 'n_tokens_mistral': 1647, 'n_tokens_neox': 1550, 'n_words': 383, 'path': 'Data_Science_Utils/Feature_Transformations/Scaling_Standardizing_Normalizing.ipynb', 'repo_name': 'shwetapurushe/data-science-world', 'size': '4377'}