<jupyter_start><jupyter_text>Standardizing Data> This chapter is all about standardizing data. Often a model will make some assumptions about the distribution or scale of your features. Standardization is a way to make your data fit these assumptions and improve the algorithm's performance. This is the Summary of lecture "Preprocessing for Machine Learning in Python", via datacamp.- toc: true - badges: true- comments: true- author: <NAME>- categories: [Python, Datacamp, Machine_Learning]- image:<jupyter_code>import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Standardizing Data- Standardization    - Preprocessing method used to transform continuous data to make it look normally distributed    - Scikit-learn models assume normally distributed data        - Log normalization        - feature Scaling- When to standardize: models    - Model in linear space    - Dataset features have high variance    - Dataset features are continuous and on different scales    - Linearity assumptions  Modeling without normalizingLet's take a look at what might happen to your model's accuracy if you try to model data without doing some sort of standardization first. Here we have a subset of the wine dataset. One of the columns, `Proline`, has an extremely high variance compared to the other columns. This is an example of where a technique like log normalization would come in handy, which you'll learn about in the next section.The scikit-learn model training process should be familiar to you at this point, so we won't go too in-depth with it. You already have a k-nearest neighbors model available (`knn`) as well as the `X` and `y` sets you need to fit and score on.<jupyter_code>wine = pd.read_csv('./dataset/wine_types.csv')
wine.head()
X = wine[['Proline', 'Total phenols', 'Hue', 'Nonflavanoid phenols']]
y = wine['Type'] 
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier()

# Split the dataset and labels into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y)

# Fit the k-nearest neighbors model to the training data
knn.fit(X_train, y_train)

# SCore the model on the test data
print(knn.score(X_test, y_test))<jupyter_output>0.6888888888888889<jupyter_text>Log normalization- Applies log transformation- Natural log using the constant $e$ (2.718)- Captures relative changes, the magnitude of change, and keeps everything in the positive space  Checking the varianceCheck the variance of the columns in the `wine` dataset.<jupyter_code>wine.describe()<jupyter_output><empty_output><jupyter_text>The `Proline` column has an extremely high variance.  Log normalization in PythonNow that we know that the `Proline` column in our wine dataset has a large amount of variance, let's log normalize it.<jupyter_code># Print out the variance of the Proline column
print(wine['Proline'].var())

# Apply the log normalization function to the Proline column
wine['Proline_log'] = np.log(wine['Proline'])

# Check the variance of the normalized Proline column
print(wine['Proline_log'].var())<jupyter_output>99166.71735542428
0.17231366191842018<jupyter_text>Scaling data for feature comparison- Features on different scales- Model with linear characteristics- Center features around 0 and transform to unit variance(1)- Transforms to approximately normal distribution  Scaling data - investigating columnsWe want to use the `Ash`, `Alcalinity of ash`, and `Magnesium` columns in the `wine` dataset to train a linear model, but it's possible that these columns are all measured in different ways, which would bias a linear model. Using `describe()` to return descriptive statistics about this dataset, which of the following statements are true about the scale of data in these columns?<jupyter_code>wine[['Ash', 'Alcalinity of ash', 'Magnesium']].describe()<jupyter_output><empty_output><jupyter_text>Scaling data - standardizing columnsSince we know that the `Ash`, `Alcalinity of ash`, and `Magnesium` columns in the `wine` dataset are all on different scales, let's standardize them in a way that allows for use in a linear model.<jupyter_code>from sklearn.preprocessing import StandardScaler

# Create the scaler
ss = StandardScaler()

# Take a subset of the DataFrame you want to scale
wine_subset = wine[['Ash', 'Alcalinity of ash', 'Magnesium']]

print(wine_subset.iloc[:3])

# Apply the scaler to the DataFrame subset
wine_subset_scaled = ss.fit_transform(wine_subset)

print(wine_subset_scaled[:3])<jupyter_output>Ash  Alcalinity of ash  Magnesium
0  2.43               15.6        127
1  2.14               11.2        100
2  2.67               18.6        101
[[ 0.23205254 -1.16959318  1.91390522]
 [-0.82799632 -2.49084714  0.01814502]
 [ 1.10933436 -0.2687382   0.08835836]]<jupyter_text>Standardized data and modeling  KNN on non-scaled dataLet's first take a look at the accuracy of a K-nearest neighbors model on the `wine` dataset without standardizing the data. The `knn` model as well as the `X` and `y` data and labels sets have been created already. Most of this process of creating models in scikit-learn should look familiar to you.<jupyter_code>wine = pd.read_csv('./dataset/wine_types.csv')

X = wine.drop('Type', axis=1)
y = wine['Type'] 

knn = KNeighborsClassifier()
# Split the dataset and labels into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y)

# Fit the k-nearest neighbors model to the training data
knn.fit(X_train, y_train)

# Score the model on the test data
print(knn.score(X_test, y_test))<jupyter_output>0.7555555555555555<jupyter_text>KNN on scaled dataThe accuracy score on the unscaled wine dataset was decent, but we can likely do better if we scale the dataset. The process is mostly the same as the previous exercise, with the added step of scaling the data.<jupyter_code>knn = KNeighborsClassifier()

# Create the scaling method
ss = StandardScaler()

# Apply the scaling method to the dataset used for modeling
X_scaled = ss.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

# Fit the k-nearest neighbors model to the training data.
knn.fit(X_train, y_train)

# Score the model on the test data
print(knn.score(X_test, y_test))<jupyter_output>0.9555555555555556
{'fraction_non_alphanumeric': 0.07293149818066762, 'fraction_numerical': 0.0322733744660655, 'mean_word_length': 4.930581613508442, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 3, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '546716', 'license': 'Apache-2.0', 'n_tokens_mistral': 1993, 'n_tokens_neox': 1815, 'n_words': 778, 'path': '_notebooks/2020-07-09-03-Standardizing-Data.ipynb', 'repo_name': 'AntonovMikhail/chans_jupyter', 'size': '6328'}