<jupyter_start><jupyter_text>1. Outlier detection- An outlier is an observation that is unlike the other observations. - They are rare, distinct, or do not fit in some way- Outliers can be measurement or input error, data corruption ot true outlier observationIdentifying outliers and bad data one of the most difficult parts of data cleanup, and it takes time to get right.**This task need to be done cautiously**. 1.1 Standard Deviation Method**3x standard deviations ($\sigma$) from the mean ($\mu$)** is a common cut-off in practice for identifying outliers in a Gaussian or Gaussian-like distribution.<jupyter_code>from numpy.random import seed 
from numpy.random import randn 
from numpy import mean
from numpy import std
import seaborn as sns

seed(1)
data = 5 * randn(1000) + 50
# sns.distplot(data)
sns.histplot(data)

# calculate summary statistics
data_mean, data_std = mean(data), std(data)
print('len=', len(data), 'mean=', data_mean, 'std=', data_std)

# define outliers
cut_off = data_std * 3
lower, upper = data_mean - cut_off, data_mean + cut_off 
print('cutoff=', cut_off, 'lower=', lower, 'upper=', upper)

# identify outliers
outliers = [x for x in data if x < lower or x > upper] 
print('Identified outliers: %d' % len(outliers))

# remove outliers
outliers_removed = [x for x in data if x >= lower and x <= upper]
print('Non-outlier observations: %d' % len(outliers_removed))<jupyter_output>len= 1000 mean= 50.194062380798016 std= 4.9050206696610585
cutoff= 14.715062008983175 lower= 35.47900037181484 upper= 64.9091243897812
Identified outliers: 4
Non-outlier observations: 996<jupyter_text>1.2 Interquartile Range MethodLimits on the sample values that are a factor $k=1.5$ of the $IQR$ below the $25$ percentile or above $75$ percentile.<jupyter_code>from numpy.random import seed
from numpy.random import randn 
from numpy import percentile

seed(1)
data = 5 * randn(1000) + 50
# sns.distplot(data)
sns.histplot(data)
print('len=', len(data))
      
# calculate interquartile range
q25, q75 = percentile(data, 25), percentile(data, 75)
iqr = q75 - q25
print('Percentiles: 25th=%.3f, 75th=%.3f, IQR=%.3f' % (q25, q75, iqr)) 

# calculate the outlier cutoff
cut_off = iqr * 1.5
lower, upper = q25 - cut_off, q75 + cut_off
print('cutoff=', cut_off, 'lower=', lower, 'upper=', upper)

# identify outliers
outliers = [x for x in data if x < lower or x > upper] 
print('Identified outliers: %d' % len(outliers))

# remove outliers
outliers_removed = [x for x in data if x >= lower and x <= upper] 
print('Non-outlier observations: %d' % len(outliers_removed))<jupyter_output>len= 1000
Percentiles: 25th=46.999, 75th=53.520, IQR=6.521
cutoff= 9.781194643044664 lower= 37.21800319685014 upper= 63.30118891163591
Identified outliers: 9
Non-outlier observations: 991<jupyter_text>2. Imputation of missing values  Replace 0 by numpy NaN in several columns<jupyter_code>import numpy as np
from pandas import read_csv

dataset = read_csv('data/pima-indians-diabetes.csv', header=None) 
print(dataset.head(20))

num_missing = (dataset[[1,2,3,4,5,6,7]] == 0).sum()
print(num_missing)

dataset[[1,2,3,4,5]] = dataset[[1,2,3,4,5]].replace(0, np.nan)
print(dataset.isnull().sum())
print(dataset.head(20))
print(dataset.shape)<jupyter_output>0    1   2   3    4     5      6   7  8
0    6  148  72  35    0  33.6  0.627  50  1
1    1   85  66  29    0  26.6  0.351  31  0
2    8  183  64   0    0  23.3  0.672  32  1
3    1   89  66  23   94  28.1  0.167  21  0
4    0  137  40  35  168  43.1  2.288  33  1
5    5  116  74   0    0  25.6  0.201  30  0
6    3   78  50  32   88  31.0  0.248  26  1
7   10  115   0   0    0  35.3  0.134  29  0
8    2  197  70  45  543  30.5  0.158  53  1
9    8  125  96   0    0   0.0  0.232  54  1
10   4  110  92   0    0  37.6  0.191  30  0
11  10  168  74   0    0  38.0  0.537  34  1
12  10  139  80   0    0  27.1  1.441  57  0
13   1  189  60  23  846  30.1  0.398  59  1
14   5  166  72  19  175  25.8  0.587  51  1
15   7  100   0   0    0  30.0  0.484  32  1
16   0  118  84  47  230  45.8  0.551  31  1
17   7  107  74   0    0  29.6  0.254  31  1
18   1  103  30  38   83  43.3  0.183  33  0
19   1  115  70  30   96  34.6  0.529  32  1
1      5
2     35
3    227
4    374
5     11
6      0
7[...]<jupyter_text>3.1 Remove rows with empty values<jupyter_code>dataset.dropna(inplace=True)
print(dataset.shape)<jupyter_output>(392, 9)<jupyter_text>3.2 Statistical Imputation **Simple imputer**<jupyter_code>import numpy as np
from sklearn.impute import SimpleImputer
data = [[7, 2, 3], [4, np.nan, 6], [10, 5, 9]]
print(data)

imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean')
imp_mean.fit(data)

X = [[np.nan, 2, 3], [4, np.nan, 6], [10, np.nan, 9]]
print(imp_mean.transform(X))<jupyter_output>[[7, 2, 3], [4, nan, 6], [10, 5, 9]]
[[ 7.   2.   3. ]
 [ 4.   3.5  6. ]
 [10.   3.5  9. ]]<jupyter_text>**Simple imputer with horse-colic dataset**<jupyter_code>import numpy as np
from numpy import isnan
from pandas import read_csv
from sklearn.impute import SimpleImputer

# load dataset
dataframe = read_csv('data/horse-colic.csv', header=None, na_values='?') 

# split into input and output elements
data = dataframe.values
ix = [i for i in range(data.shape[1]) if i != 23]
X, y = data[:, ix], data[:, 23]

# summarize total missing
print('Missing: %d' % sum(isnan(X).flatten())) 

# define imputer
imputer = SimpleImputer(strategy='mean')

# fit on the dataset
imputer.fit(X)

# transform the dataset
X_trans = imputer.transform(X)

# summarize total missing
print('Missing: %d' % sum(isnan(X_trans).flatten()))<jupyter_output>Missing: 1605
Missing: 0<jupyter_text>**kNN imputation**<jupyter_code>from numpy import isnan
from pandas import read_csv
from sklearn.impute import KNNImputer

# load dataset
dataframe = read_csv('data/horse-colic.csv', header=None, na_values='?') 

# split into input and output elements
data = dataframe.values
ix = [i for i in range(data.shape[1]) if i != 23]
X, y = data[:, ix], data[:, 23]

# summarize total missing
print('Missing: %d' % sum(isnan(X).flatten()))

# define imputer
imputer = KNNImputer()

# fit on the dataset
imputer.fit(X)

# transform the dataset
Xtrans = imputer.transform(X)

# summarize total missing
print('Missing: %d' % sum(isnan(Xtrans).flatten()))<jupyter_output>Missing: 1605
Missing: 0<jupyter_text>3. Data Transformations**ML models learn a mapping from input variables to an output variable** 3.1 EncodingURL https://contrib.scikit-learn.org/category_encoders/**Poznamka: Python 3.8 FutureWarning is OK for the library category_encoders**  3.1.1 Ordinal Encoding or Label Encoding<jupyter_code>import pandas as pd
import category_encoders as ce

data = pd.DataFrame({
    'city' : ['delhi', 'hyderabad', 'delhi', 'delhi', 'gurgaon', 'hyderabad']
})

# create an object of the OrdinalEncoding
ce_ordinal = ce.OrdinalEncoder(cols=['city'])

# fit and transform and you will get the encoded data
ce_ordinal.fit_transform(data)<jupyter_output>C:\Users\misko\AppData\Local\Programs\Python\Python39\lib\site-packages\category_encoders\utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>3.1.2 One Hot Encoding<jupyter_code>import pandas as pd
import category_encoders as ce

data = pd.DataFrame({
    'gender' : ['M', 'F', 'M', 'F', 'F']
})

# create an object of the OneHotEncoder
ce_OHE = ce.OneHotEncoder(cols=['gender'])

# fit and transform and you will get the encoded data
ce_OHE.fit_transform(data)<jupyter_output>C:\Users\misko\AppData\Local\Programs\Python\Python39\lib\site-packages\category_encoders\utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>3.1.3 Binary Encoding<jupyter_code>import pandas as pd
import category_encoders as ce

# make some data
data = pd.DataFrame({
    'class' : ['a', 'b', 'a', 'b', 'd', 'e', 'd', 'f', 'g', 'h', 'h', 'k', 'h', 'i', 's', 'p', 'z']})

# create object of BinaryEncoder
ce_binary = ce.BinaryEncoder(cols = ['class'])

# fit and transform and you will get the encoded data
ce_binary.fit_transform(data)<jupyter_output>C:\Users\misko\AppData\Local\Programs\Python\Python39\lib\site-packages\category_encoders\utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>3.1.4 BaseN Encoding<jupyter_code>import pandas as pd
import category_encoders as ce

# make some data
data = pd.DataFrame({
    'class' : ['a', 'b', 'a', 'b', 'd', 'e', 'd', 'f', 'g', 'h', 'h', 'k', 'h', 'i', 's', 'p', 'z']})

# create an object of the BaseNEncoder
ce_baseN4 = ce.BaseNEncoder(cols=['class'], base=4)

# fit and transform and you will get the encoded data
ce_baseN4.fit_transform(data)<jupyter_output>C:\Users\misko\AppData\Local\Programs\Python\Python39\lib\site-packages\category_encoders\utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>3.1.5 Hashing (md5)<jupyter_code>import pandas as pd
import category_encoders as ce

data = pd.DataFrame({
    'color' : ['Yellow', 'Black', 'Green', 'Blue', 'Blue', 'Green', 'Black', 'Blue']
})

# create an object of the HashingEncoder
ce_HE = ce.HashingEncoder(cols=['color'],n_components=5)

# fit and transform and you will get the encoded data
ce_HE.fit_transform(data)<jupyter_output>C:\Users\misko\AppData\Local\Programs\Python\Python39\lib\site-packages\category_encoders\utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>3.1.6 Target Encoding<jupyter_code>import pandas as pd
import category_encoders as ce

data = pd.DataFrame({
    'color' : ['Blue', 'Black', 'Black','Blue', 'Blue'],
    'outcome' : [1,      2,        1,     1,      2,]
})

# column to perform encoding
X = data['color']
Y = data['outcome']

# create an object of the TargetEncoder
ce_TE = ce.TargetEncoder(cols=['color'])

# fit and transform and you will get the encoded data
ce_TE.fit_transform(X,Y)<jupyter_output>C:\Users\misko\AppData\Local\Programs\Python\Python39\lib\site-packages\category_encoders\utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>3.1.7 Leave One Out<jupyter_code>import pandas as pd
import category_encoders as ce

data = pd.DataFrame({
    'color' : ['Blue', 'Black', 'Black','Blue', 'Blue'],
    'outcome' : [2,      1,        1,     1,      2]
})

# column to perform encoding
X = data['color']
Y = data['outcome']

# create an object of the TargetEncoder
ce_TE = ce.LeaveOneOutEncoder(cols=['color'])

# fit and transform and you will get the encoded data
ce_TE.fit_transform(X,Y)<jupyter_output>C:\Users\misko\AppData\Local\Programs\Python\Python39\lib\site-packages\category_encoders\utils.py:21: FutureWarning: is_categorical is deprecated and will be removed in a future version.  Use is_categorical_dtype instead
  elif pd.api.types.is_categorical(cols):<jupyter_text>3.2 Data scaling- The scale and distribution of the data drawn from the domain may be different for each variable- This can lead to unstable or poor performance of models- This can lead to increasing sensitivity to inputs- This can lead to higher generalization error**3.2.1 Data Normalization** $x_{normalization}=\frac{x-x_{min}}{x_{max} - x_{min}}$URL https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html<jupyter_code>from numpy import asarray
from sklearn.preprocessing import MinMaxScaler 

# define data
data = asarray([[100, 0.001],
[8, 0.05], [50, 0.005], [88, 0.07], [4, 0.1]])
print(data)

# define min max scaler
scaler = MinMaxScaler()

# transform data
scaled = scaler.fit_transform(data) 
print(scaled)<jupyter_output>[[1.0e+02 1.0e-03]
 [8.0e+00 5.0e-02]
 [5.0e+01 5.0e-03]
 [8.8e+01 7.0e-02]
 [4.0e+00 1.0e-01]]
[[1.         0.        ]
 [0.04166667 0.49494949]
 [0.47916667 0.04040404]
 [0.875      0.6969697 ]
 [0.         1.        ]]<jupyter_text>**3.2.2 Data Standardization** $x_{standardized} = \frac{x -\mu}{\sigma}$where - $\mu$ is the mean  of $x$- $\sigma$ is the standard deviation of $x$URL https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html<jupyter_code>from numpy import asarray
from sklearn.preprocessing import StandardScaler 

# define data
data = asarray([[100, 0.001], [8, 0.05], [50, 0.005], [88, 0.07], [4, 0.1]])
print(data)

# define standard scaler
scaler = StandardScaler()

# transform data
scaled = scaler.fit_transform(data) 
print(scaled)<jupyter_output>[[1.0e+02 1.0e-03]
 [8.0e+00 5.0e-02]
 [5.0e+01 5.0e-03]
 [8.8e+01 7.0e-02]
 [4.0e+00 1.0e-01]]
[[ 1.26398112 -1.16389967]
 [-1.06174414  0.12639634]
 [ 0.         -1.05856939]
 [ 0.96062565  0.65304778]
 [-1.16286263  1.44302493]]<jupyter_text>**3.2.3 Robust Scaling**Robust scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile). $x_{robust} = \frac{x - median}{p_{75} - p_{25}}$where - $median$ is the median of $x$- $p_{25}$ and $p_{75}$ are 25th and 75th interquartile range values (IQR).URL https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html<jupyter_code>from sklearn.preprocessing import RobustScaler

# define data
X = [[ 1., -2.,  2.],
     [ -2.,  1.,  3.],
     [ 4.,  1., -2.]]
print(X)

# define robust scaler
scaler = RobustScaler()

# transform data
scaled = RobustScaler().fit_transform(X)
print(scaled)<jupyter_output>[[1.0, -2.0, 2.0], [-2.0, 1.0, 3.0], [4.0, 1.0, -2.0]]
[[ 0.  -2.   0. ]
 [-1.   0.   0.4]
 [ 1.   0.  -1.6]]<jupyter_text>3.3 Making data distributions more Gaussian (normal)Many ML algorithms perform better or when features are on a relatively similar to normally (Gaussian) distributed. 3.3.1 Power Transformer - Replacing the data with the log, square root, or inverse to remove skew- Yeo-Johnson transform (default): works with positive and negative values- Box-Cox transform: only works with strictly positive values- λ = −1.0 is a reciprocal transform.- λ = −0.5 is a reciprocal square root transform.  - λ = 0.0 is a log transform.- λ = 0.5 is a square root transform.- λ = 1.0 is no transform.**Random data**<jupyter_code>from numpy import exp
from numpy.random import randn
from sklearn.preprocessing import PowerTransformer
from matplotlib import pyplot

# generate gaussian data sample
data = randn(1000)

# add a skew to the data distribution
data = exp(data)

# histogram of the raw data with a skew
pyplot.hist(data, bins=25)

# reshape data to have rows and columns
data = data.reshape((len(data),1))

# power transform the raw data
power = PowerTransformer(method='yeo-johnson', standardize=True) 
data_trans = power.fit_transform(data)
 
# histogram of the transformed data
pyplot.hist(data_trans, bins=25)<jupyter_output><empty_output><jupyter_text>**Sonar data**<jupyter_code>from matplotlib import pyplot as plt
from pandas import read_csv
from pandas import DataFrame
from sklearn.preprocessing import PowerTransformer 

plt.rcParams['figure.figsize'] = [15, 10]

# Load dataset
dataset = read_csv('data/sonar.csv', header=None)

# retrieve just the numeric input values
data = dataset.values[:, :-1]

# perform a yeo-johnson transform of the dataset 
pt = PowerTransformer(method='yeo-johnson')
data = pt.fit_transform(data)

# convert the array back to a dataframe
dataset = DataFrame(data)

# histograms of the variables
fig = dataset.hist()<jupyter_output><empty_output><jupyter_text>3.3.2 Quantile Transformer- Transform features using quantiles information.- This method transforms the features to follow a uniform or a normal distribution. - Therefore, for a given feature, this transformation tends to spread out the most frequent values. - It also reduces the impact of (marginal) outliers<jupyter_code>from matplotlib import pyplot
import numpy as np
from sklearn.preprocessing import QuantileTransformer

rng = np.random.RandomState(0)
X = np.sort(rng.normal(loc=0.5, scale=0.25, size=(100, 1)), axis=0)
pyplot.hist(X, bins=10)

qt = QuantileTransformer(n_quantiles=10, random_state=0)
X_t = qt.fit_transform(X)
pyplot.hist(X_t, bins=10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12482048827190043, 'fraction_numerical': 0.07192436572522738, 'mean_word_length': 4.2789008212255215, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 68, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '199540', 'license': 'Apache-2.0', 'n_tokens_mistral': 6756, 'n_tokens_neox': 6046, 'n_words': 1816, 'path': 'cvicenia/tyzden-06/IAU_061_outlier_missing_transform.ipynb', 'repo_name': 'michalboros/2021-2022', 'size': '16712'}