<jupyter_start><jupyter_text>TOC __Chapter 3 - Feature improvement - cleaning datasets__1. [Import](Import)1. [Exploratory data analysis](Exploratory-data-analysis)    1. [Univariate & feature vs. target](Univariate-&-feature-vs.-target)    1. [Correlation](Correlation)    1. [Missing data](Missing-data)        1. [Remove missing data](Remove-missing-data)        1. [Impute missing data](Impute-missing-data)            1. [Pipelines - imputing and feature scaling ](Pipelines-imputing-and-feature-scaling)  Import<jupyter_code># standard libary and settings
import os
import sys
import importlib
import itertools
from io import StringIO
import warnings

warnings.simplefilter("ignore")
from IPython.core.display import display, HTML

display(HTML("<style>.container { width:95% !important; }</style>"))

# data extensions and settings
import numpy as np

np.set_printoptions(threshold=np.inf, suppress=True)
import pandas as pd

pd.set_option("display.max_rows", 500)
pd.options.display.float_format = "{:,.6f}".format

# modeling extensions
from sklearn.base import TransformerMixin, BaseEstimator
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering
from sklearn.datasets import load_boston, load_wine, load_iris, load_breast_cancer, make_blobs, make_moons
from sklearn.decomposition import PCA, LatentDirichletAllocation
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, AdaBoostClassifier, ExtraTreesClassifier, IsolationForest
from sklearn.feature_extraction.text import CounterVectorizer, TfidfTransformer, TfidfVectorizer, HashingVectorizer
from sklearn.feature_selection import f_classif, f_regression, VarianceThreshold, SelectFromModel, SelectKBest
from sklearn.impute import SimpleImputer
from sklearn.linear_model import Lasso, Ridge, ElasticNet, LinearRegression, LogisticRegression, SGDRegressor
from sklearn.metrics import precision_score, recall_score, f1_score, explained_variance_score, mean_squared_log_error, mean_absolute_error, median_absolute_error, mean_squared_error, r2_score, confusion_matrix, roc_curve, accuracy_score, roc_auc_score, homogeneity_score, completeness_score, classification_report, silhouette_samples
from sklearn.model_selection import KFold, train_test_split, GridSearchCV, StratifiedKFold, cross_val_score, RandomizedSearchCV
from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor
from sklearn.pipeline import make_pipeline, Pipeline, FeatureUnion
from sklearn.preprocessing import StandardScaler, RobustScaler, PolynomialFeatures, OrdinalEncoder, LabelEncoder, OneHotEncoder, KBinsDiscretizer, QuantileTransformer, PowerTransformer, MinMaxScaler
from sklearn.svm import SVC, SVR
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import sklearn.utils as utils

# visualization extensions and settings
import seaborn as sns
import matplotlib.pyplot as plt

import mlmachine as mlm
from prettierplot.plotter import PrettierPlot
import prettierplot.style as style

# magic functions
%matplotlib inline
# load data and inspect
pima_column_names = [
    "times_pregnant",
    "plasma_glucose_concentration",
    "diastolic_blood_pressure",
    "triceps_thickness",
    "serum_insulin",
    "bmi",
    "pedigree_function",
    "age",
    "onset_diabetes",
]

df = pd.read_csv("../misc/PimaIndians.csv", sep="|", names=pima_column_names)
df.info()
display(df.head())
# null accuracy
df["onset_diabetes"].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Exploratory data analysis   Univariate & feature vs. target<jupyter_code># preliminary EDA using MLMachine
# sns.set_style("whitegrid")
train = mlm.Machine(data=df, target=["onset_diabetes"], target_type="object")
# m.transformLabel()
train.eda_cat_target_num_feat()<jupyter_output>******************
object label encoding

0 --> 0
1 --> 1<jupyter_text>> Remarks - Notice that several features have 0 as a minimum value where a value of 0 is impossible. For example, a BMI of 0 is impossible. However, a times_pregnant value of 0 is certainly possible. So it appears that 0 is being using to indicate missing values for a few of the columns, namely bmi, plasma_glucose_concentration, diastrolic_blood_pressure, triceps_thickness and serum_insulin.  Correlation<jupyter_code># correlation heat map
p = PrettierPlot(chart_scale=20)
ax = p.make_canvas()
p.corr_heatmap(df=train.recombine_data(train.data, train.target), ax=ax)
# display correlation coefficients for target
df.corr()["onset_diabetes"]<jupyter_output><empty_output><jupyter_text>Missing data<jupyter_code># count missing values
df.isnull().sum()
# calculate summary statistics
df.describe()<jupyter_output><empty_output><jupyter_text>> Remarks - Several features, such as 'BMI' and 'diastolic_blood_pressure' have a minimum value of zero where such a value would be impossible. This leads us to believe '0' is being used to indicate missing values in several of the columns<jupyter_code># replace nulls with none
columns = [
    "serum_insulin",
    "bmi",
    "plasma_glucose_concentration",
    "diastolic_blood_pressure",
    "triceps_thickness",
]
for col in columns:
    df[col].replace([0], [None], inplace=True)
# count missing values
df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Remove missing data  __Drop all row with NA__<jupyter_code># drop all rows that have a missing value
df_dropped = df.dropna()

num_rows_lost = (df.shape[0] - df_dropped.shape[0]) / df.shape[0]
print("Proportion of data lost: {0}".format(np.round(num_rows_lost, 4)))<jupyter_output>Proportion of data lost: 0.4896<jupyter_text>__Evaluate impact on features__<jupyter_code># category distribution prior to dropping records
df["onset_diabetes"].value_counts(normalize=True)
# category distribution after dropping records
df_dropped["onset_diabetes"].value_counts(normalize=True)
# display average value in each column prior to dropping records
df.mean()
# display average value ine each column after dropping records

df_dropped.mean()
# prior change in means
(df_dropped.mean() - df.mean()) / df.mean()<jupyter_output><empty_output><jupyter_text>> Remarks - 'times_pregnant' fell by 14%, while 'pedigree_function' increased by 10%. These are pretty substantial changes to the shape of the data, which justifies the need to retain as much data as possible.  Impute missing data<jupyter_code># identify rows that have missing values
empty_plasma_index = df[df["plasma_glucose_concentration"].isnull()].index
df.loc[empty_plasma_index]["plasma_glucose_concentration"]<jupyter_output><empty_output><jupyter_text>__Impute with mean__<jupyter_code># impute missing values with column average
df["plasma_glucose_concentration"].fillna(
    df["plasma_glucose_concentration"].mean(), inplace=True
)
df.loc[empty_plasma_index]["plasma_glucose_concentration"]
# use sklearn to impute
imputer = Imputer(strategy="mean")
df_imputed = imputer.fit_transform(df)
df_imputed = pd.DataFrame(df_imputed, columns=pima_column_names)
df_imputed[:5]
# identify records with missing values
df_imputed.loc[empty_plasma_index]["plasma_glucose_concentration"]
# count number of missing values by columns
df_imputed.isnull().sum()<jupyter_output><empty_output><jupyter_text>Pipelines - imputing and feature scaling   __Standard scaling and Max/Min scaling__<jupyter_code>#  StandardScaler
knn_params = {
    "imputer__strategy": ["mean", "median"],
    "classify__n_neighbors": [1, 2, 3, 4, 5, 6, 7],
}

mean_impute_standardize = Pipeline(
    [
        ("imputer", SimpleImputer()),
        ("standardize", StandardScaler()),
        ("classify", KNeighborsClassifier()),
    ]
)
X = df.drop("onset_diabetes", axis=1)
y = df["onset_diabetes"]

grid = GridSearchCV(mean_impute_standardize, knn_params)
grid.fit(X, y)

print(grid.best_score_, grid.best_params_)
# MinMaxScaler
knn_params = {
    "imputer__strategy": ["mean", "median"],
    "classify__n_neighbors": [1, 2, 3, 4, 5, 6, 7],
}

mean_impute_standardize = Pipeline(
    [
        ("imputer", SimpleImputer()),
        ("standardize", MinMaxScaler()),
        ("classify", KNeighborsClassifier()),
    ]
)
X = df.drop("onset_diabetes", axis=1)
y = df["onset_diabetes"]

grid = GridSearchCV(mean_impute_standardize, knn_params)
grid.fit(X, y)

print(grid.best_score_, grid.best_params_)<jupyter_output>0.74609375 {'classify__n_neighbors': 4, 'imputer__strategy': 'mean'}<jupyter_text>__Row normalization__Makes every row (vector) have the same norm, or length. For example, the L2 norm for rows in the pima dataset is calculated as:$$\lVert{x}\lVert = \sqrt{(x_1^2 + x_2^2 + x_3^2 + x_4^2 + x_5^2 + x_6^2 + x_7^2 + x_8^2)}$$because there are 8 features, not including the target. This is espeically useful when working with text data or clustering algorithms.<jupyter_code># compute average nomr
np.sqrt((df ** 2).sum(axis=1)).mean()
# normalize rows
normalize = preprocessing.Normalizer()

dfN_norm = pd.DataFrame(normalize.fit_transform(df_imputed), columns=pima_column_names)
np.sqrt((dfN_norm ** 2).sum(axis=1)).mean()
# row normalization
knn_params = {
    "imputer__strategy": ["mean", "median"],
    "classify__n_neighbors": [1, 2, 3, 4, 5, 6, 7],
}

mean_impute_standardize = Pipeline(
    [
        ("imputer", SimpleImputer()),
        ("normalize", preprocessing.Normalizer()),
        ("classify", KNeighborsClassifier()),
    ]
)
X = df.drop("onset_diabetes", axis=1)
y = df["onset_diabetes"]

grid = GridSearchCV(mean_impute_standardize, knn_params)
grid.fit(X, y)

print(grid.best_score_, grid.best_params_)<jupyter_output>0.6822916666666666 {'classify__n_neighbors': 6, 'imputer__strategy': 'mean'}
{'fraction_non_alphanumeric': 0.10587749199462865, 'fraction_numerical': 0.012292118582791034, 'mean_word_length': 6.5054263565891475, 'pattern_counts': {'":': 6, '<': 43, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '131849', 'license': 'MIT', 'n_tokens_mistral': 3355, 'n_tokens_neox': 3127, 'n_words': 778, 'path': 'ch03_Feature_improvement_cleaning_datasets.ipynb', 'repo_name': 'petersontylerd/feature-engineering-made-easy', 'size': '9681'}