<jupyter_start><jupyter_text>One hot encoding* Just like imputation, all methods of categorical encoding should be performed over the training set, and then propagated to the test set.* Why?    * Because these methods will "learn" patterns from the train data, and therefore you want to avoid leaking information and overfitting. But more importantly, because we don't know whether in future / live data, we will have all the categories present in the train data, or if there will be more or less categories. Therefore, we want to anticipate this uncertainty by setting the right processes right from the start. We want to create transformers that learn the categories from the train set, and used those learned categories to create the dummy variables in both train and test sets.<jupyter_code>df = pd.read_csv('titanic.csv', usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].str[0] # capture only the first letter of the cabin
df.head()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
X_train.nunique()
X_test.nunique()<jupyter_output><empty_output><jupyter_text>__One hot encoding with pandas__**pd.get_dummies can handle missing values (NAN)*** Advantages    * quick    * returns pandas dataframe    * returns feature names for the dummy variables* Limitations of pandas:    * it does not preserve information from train data to propagate to test data    * In practice, we shouldn't be using get-dummies in our machine learning pipelines. It is however useful, for a quick data exploration.    * The train set contains 13 dummy features, whereas the test set contains 12 features. This occurred because there was no category T in cabin in the test set. This will cause problems if training and scoring models with scikit-learn, because predictors require train and test sets to be of the same shape.<jupyter_code>pd.get_dummies(data=X_train, prefix="titanic", prefix_sep="-", 
               drop_first=False, 
               dummy_na=True # # we can add an additional dummy variable to indicate missing data
              ).head()
pd.get_dummies(data=X_test, prefix="titanic", prefix_sep="-", 
               drop_first=False, 
               dummy_na=True # # we can add an additional dummy variable to indicate missing data
              ).head()<jupyter_output><empty_output><jupyter_text>__One hot encoding with Scikit-learn__* [OneHotEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html)<jupyter_code>X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer, make_column_transformer

features = X.columns

# preprocessor = ColumnTransformer(transformers=[
#     ('OneHotEncoder', 
#      OneHotEncoder(categories='auto', 
#                    drop=None, 
#                    sparse=False, 
#                    handle_unknown='ignore'  # helps deal with rare labels
#                   ),
#      features),
# ], remainder="drop")

preprocessor = make_column_transformer(
    (OneHotEncoder(categories='auto', 
                   drop=None, 
                   sparse=False, 
                   handle_unknown='ignore'  # helps deal with rare labels in the X_test
                  ),
     features), remainder="drop")
X_train = preprocessor.fit_transform(X_train)
X_test = preprocessor.transform(X_test)
preprocessor
preprocessor.named_transformers_
preprocessor.named_transformers_["onehotencoder"]
preprocessor.named_transformers_["onehotencoder"].categories_
preprocessor.named_transformers_["onehotencoder"].get_feature_names()
X_train
pd.DataFrame(data=X_train, columns=preprocessor.named_transformers_["onehotencoder"].get_feature_names()).head()<jupyter_output><empty_output><jupyter_text>__One hot encoding with feature_engine__* quick* returns dataframe* returns feature names* allows to select features to encode[fe_OneHotEncoder](https://feature-engine.readthedocs.io/en/1.1.x/encoding/OneHotEncoder.htmlapi-reference)<jupyter_code>X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from sklearn.impute import SimpleImputer
from feature_engine.encoding import OneHotEncoder as fe_OneHotEncoder
from sklearn.compose import ColumnTransformer, make_column_transformer
from sklearn.pipeline import Pipeline, make_pipeline

features = X.columns.tolist()

pipeline = make_pipeline(
    SimpleImputer(strategy="constant", fill_value="Missing"), # return a numpy instead of dataframe
    fe_OneHotEncoder(top_categories=None, drop_last=False, 
#                    variables=features # Specify features name. Since it numpy array from Simputer()
#                                       # and we want to encode all features, no need to use variables=features
                    )
)
X_train = pipeline.fit_transform(X_train)
X_test = pipeline.transform(X_test)
pipeline
pipeline.named_steps
pipeline.named_steps["onehotencoder"].encoder_dict_
pipeline.named_steps["onehotencoder"].variables_
X_train.head()
X_test.head()<jupyter_output><empty_output><jupyter_text>__Allows to select features to encode__<jupyter_code>X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from feature_engine.encoding import OneHotEncoder as fe_OneHotEncoder

features = X.columns.to_list()
enc = fe_OneHotEncoder(top_categories=None, drop_last=False, variables=features) # Allows to select features to encode

X_train = enc.fit_transform(X_train.fillna('Missing'))
X_test = enc.transform(X_test.fillna('Missing'))
enc
enc.encoder_dict_
enc.variables_
X_train.head()
X_test.head()<jupyter_output><empty_output><jupyter_text>One Hot Encoding of Frequent Categories* Performing one hot encoding, only considering the most frequent categories. Often, categorical variables show a few dominating categories while the remaining labels add little information. Therefore, OHE of top categories is a simple and useful technique.<jupyter_code>df = pd.read_csv('houseprice.csv', usecols=['Neighborhood', 'Exterior1st', 'Exterior2nd', 'SalePrice'])
df.head()
for col in df.columns:
    print(f"{col}:  {len(df[col].unique())} labels")
X = df.drop(columns=["SalePrice"])
y = df["SalePrice"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, 
#                                                     stratify=y # Not a classification problem
                                                   )
# It is important to select the top or most frequent categories based of the train data. Then, we will use those top 
# categories to encode the variables in the test data as well

from feature_engine.encoding import OneHotEncoder as fe_OneHotEncoder

features = ['Neighborhood', 'Exterior1st', 'Exterior2nd']

# If the argument variables=None, then the encoder will automatically identify all categorical variables. Is that not sweet?
# The encoder will not encode numerical variables. So if some of your numerical variables are in fact categories, 
# you will need to re-cast them as object before using the encoder.

enc = fe_OneHotEncoder(top_categories=10,  # you can change this value to select more or less variables
                       drop_last=False, variables=features)

X_train = enc.fit_transform(X_train)
X_test = enc.transform(X_test)
enc
enc.encoder_dict_
enc.variables_
X_train.head()
X_test.head()<jupyter_output><empty_output><jupyter_text>Label / Ordinal("arbitrary", "ordered") / Integer encoding* Integer encoding consist in replacing the categories by digits from 1 to n (or 0 to n-1, depending the implementation), where n is the number of distinct categories of the variable. The numbers are assigned arbitrarily. This encoding method allows for quick benchmarking of machine learning models. Integer encoding is better suited for non-linear methods which are able to navigate through the arbitrarily assigned digits to try and find patters that relate them to the target.<jupyter_code>df = pd.read_csv('titanic.csv', usecols=[item.title() for item in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].str[0] # capture only the first letter of the cabin
df.head()
df.isna().mean()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
# Must fill "Missing" before LabelEncoder
X_train = X_train.fillna('Missing')
X_test = X_test.fillna('Missing')
# LabelEncoder for y
# Unfortunately, the LabelEncoder from sklearn works one variable at the time.

from sklearn.preprocessing import LabelEncoder

le_y = LabelEncoder()
y_train = le_y.fit_transform(y_train)
y_test = le_y.transform(y_test)
le_y.classes_<jupyter_output><empty_output><jupyter_text>[OrdinalEncoder vs LabelEncoder](https://datascience.stackexchange.com/questions/39317/difference-between-ordinalencoder-and-labelencoder)* OrdinalEncoder is for 2D data with the shape (n_samples, n_features)* LabelEncoder is for 1D data with the shape (n_samples,)<jupyter_code># OrdinalEncoder for X

from sklearn.preprocessing import OrdinalEncoder

oe = OrdinalEncoder() # only accept input as dataframe
X_train["Sex"] = oe.fit_transform(X_train["Sex"].to_frame())
X_test["Sex"] = oe.transform(X_test["Sex"].to_frame())

# fe_enc_arbitrary for X, fe_enc_ordered for X

from feature_engine.encoding import OrdinalEncoder as fe_OrdinalEncoder

fe_enc_arbitrary = fe_OrdinalEncoder(encoding_method='arbitrary', variables=['Cabin'])
X_train = fe_enc_arbitrary.fit_transform(X=X_train, y=None) # y=None
X_test = fe_enc_arbitrary.transform(X=X_test)

fe_enc_ordered = fe_OrdinalEncoder(encoding_method='ordered', variables=['Embarked'])
# monotonic relationship between the features and the target.
X_train = fe_enc_ordered.fit_transform(X=X_train, y=y_train) # y=y_train
X_test = fe_enc_ordered.transform(X=X_test)
X_train.head()
X_test.head()
oe.categories_
fe_enc_arbitrary.encoder_dict_
fe_enc_arbitrary.variables_
fe_enc_ordered.encoder_dict_
fe_enc_ordered.variables_<jupyter_output><empty_output><jupyter_text>Count_or_frequency_encoding* In count encoding we replace the categories by the count of the observations that show that category in the dataset. Similarly, we can replace the category by the frequency or percentage of observations in the dataset.<jupyter_code>df = pd.read_csv('houseprice.csv', usecols=['Neighborhood', 'Exterior1st', 'Exterior2nd', 'SalePrice'])
df.head()
# When doing count transformation of categorical variables, it is important to calculate the count 
# (or frequency = count / total observations) over the training set, and then use those numbers to 
# replace the labels in the test set.

X = df.drop(columns=["SalePrice"])
y = df["SalePrice"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, 
#                                                     stratify=y # Not a classification problem
                                                   )
from feature_engine.encoding import CountFrequencyEncoder

frequency_enc = CountFrequencyEncoder(
    encoding_method='frequency', # to do count ==> encoding_method='count'
    variables=['Neighborhood', 'Exterior1st', 'Exterior2nd']
)

X_train = frequency_enc.fit_transform(X_train)
X_test = frequency_enc.transform(X_test)
X_train.head()
X_test.head()
frequency_enc.encoder_dict_
frequency_enc.variables_<jupyter_output><empty_output><jupyter_text>Mean or target encoding* Create a monotonic relationship between the variable and the target, therefore suitable for linear models [fe_MeanEncoder](https://feature-engine.readthedocs.io/en/1.1.x/encoding/MeanEncoder.html)<jupyter_code>df = pd.read_csv('titanic.csv',  usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].astype(str).str[0]
df["Embarked"] = df["Embarked"].fillna(value="Missing")
df.head()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
# from sklearn.impute import SimpleImputer

# imputer = SimpleImputer(strategy="constant", fill_value="Missing") # return a numpy instead of dataframe
# X_train = imputer.fit_transform(X_train)
# X_test = imputer.transform(X_test)

from feature_engine.encoding import MeanEncoder as fe_MeanEncoder

mean_enc = fe_MeanEncoder(variables=['Cabin', 'Sex', 'Embarked']) # make sure no NaN in Features
X_train = mean_enc.fit_transform(X_train, y_train)
X_test = mean_enc.transform(X_test)
X_test.head()
mean_enc.encoder_dict_
mean_enc.variables_<jupyter_output><empty_output><jupyter_text>[TargetEncoder](https://contrib.scikit-learn.org/category_encoders/targetencoder.html)<jupyter_code>df = pd.read_csv('titanic.csv',  usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].astype(str).str[0]
# df["Embarked"] = df["Embarked"].fillna(value="Missing")
df.head()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from category_encoders.target_encoder import TargetEncoder

mean_enc = TargetEncoder(handle_missing="value", handle_unknown="value") # can handle NaN in Features
X_train = mean_enc.fit_transform(X_train, y_train)
X_test = mean_enc.transform(X_test)
X_test.head()
mean_enc.get_feature_names()
mean_enc.get_params()<jupyter_output><empty_output><jupyter_text>Probability Ratio Encoding* These encoding is suitable for **binary classification** problems only.. * If p(0) = 0 or p(1) = 0 for the ratio encoder, the encoder will return an error.<jupyter_code>df = pd.read_csv('titanic.csv',  usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].astype(str).str[0]
df["Embarked"] = df["Embarked"].fillna(value="Missing")
# df.dropna(subset=['Embarked'], inplace=True)
df.head()
# If p(0) = 0 or p(1) = 0 for the ratio encoder, the encoder will return an error.
df.iloc[829, 0] = 0
df[df["Embarked"] == "Missing"]
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from feature_engine.encoding import PRatioEncoder

ratio_enc = PRatioEncoder(encoding_method='ratio', 
                          variables=['Cabin', 'Sex', 'Embarked']) # make sure no NaN in Features

X_train = ratio_enc.fit_transform(X_train, y_train)
X_test = ratio_enc.transform(X_test)
X_test.head()
ratio_enc.encoder_dict_
ratio_enc.variables_<jupyter_output><empty_output><jupyter_text>Weight of Evidence[fe_WoEEncoder](https://feature-engine.readthedocs.io/en/1.1.x/encoding/WoEEncoder.html)<jupyter_code>df = pd.read_csv('titanic.csv',  usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].astype(str).str[0] 
# if any of the terms in the weight of evidence calculation is 0, 
# the log of 0 is not defined, so this transformer will raise an error.
df = df[df['Cabin'] != 'T']
df.dropna(subset=['Embarked'], inplace=True)
df.head()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from feature_engine.encoding import WoEEncoder as fe_WoEEncoder

woe_enc = fe_WoEEncoder(variables=['Cabin', 'Sex', 'Embarked']) # make sure no NaN in Features
X_train = woe_enc.fit_transform(X_train, y_train)
X_test = woe_enc.transform(X_test)
X_test.head()
woe_enc.encoder_dict_
woe_enc.variables_<jupyter_output><empty_output><jupyter_text>[WOEEncoder](https://contrib.scikit-learn.org/category_encoders/woe.html)<jupyter_code>df = pd.read_csv('titanic.csv',  usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].astype(str).str[0] 
df.head()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from category_encoders.woe import WOEEncoder


woe_enc = WOEEncoder(cols=['Cabin', 'Sex', 'Embarked']) # can handle NaN in Features
X_train = woe_enc.fit_transform(X_train, y_train)
X_test = woe_enc.transform(X_test)
X_test.head()
woe_enc.get_feature_names()
woe_enc.get_params()
woe_enc.mapping<jupyter_output><empty_output><jupyter_text>Rare Label Encoding[fe_RareLabelEncoder](https://feature-engine.readthedocs.io/en/1.1.x/encoding/RareLabelEncoder.html)<jupyter_code>df = pd.read_csv('houseprice.csv')

X = df.drop(columns=["SalePrice"])
y = df["SalePrice"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, 
#                                                     stratify=y # Not a classification problem
                                                   )
from feature_engine.encoding import RareLabelEncoder as fe_RareLabelEncoder

rare_enc = fe_RareLabelEncoder( # make sure no NaN in Features
    tol=0.05,  # minimal percentage to be considered non-rare
    n_categories=4, # minimal number of categories the variable should have to re-cgroup rare categories
    variables=['Neighborhood', 'Exterior1st', 'Exterior2nd',
               'MasVnrType', 'ExterQual', 'BsmtCond'] # variables to re-group
)

X_train = rare_enc.fit_transform(X_train.fillna("Missing"))
X_test = rare_enc.transform(X_test.fillna("Missing"))
X_test.head()
rare_enc.variables_
rare_enc.encoder_dict_<jupyter_output><empty_output><jupyter_text>Binary Encoding (Not often in the industry)[BinaryEncoder](https://contrib.scikit-learn.org/category_encoders/binary.html)<jupyter_code>df = pd.read_csv('titanic.csv',  usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].astype(str).str[0] 
df.head()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
from category_encoders.binary import BinaryEncoder

bin_enc = BinaryEncoder()
X_train = bin_enc.fit_transform(X_train)
X_test = bin_enc.transform(X_test)
X_test.head()
bin_enc.get_feature_names()
bin_enc.get_params()<jupyter_output><empty_output><jupyter_text>Hashing Encoding (Not often in the industry)[HashingEncoder](https://contrib.scikit-learn.org/category_encoders/hashing.html)<jupyter_code>df = pd.read_csv('titanic.csv',  usecols=[col.title() for col in ['sex', 'embarked', 'cabin', 'survived']])
df['Cabin'] = df['Cabin'].astype(str).str[0] 
df.head()
X = df.drop(labels=["Survived"], axis=1)
y = df["Survived"]

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, shuffle=True, stratify=y)
%%time
from category_encoders.hashing import HashingEncoder

hash_enc = HashingEncoder()
X_train = hash_enc.fit_transform(X_train)
X_test = hash_enc.transform(X_test)
X_test.head()
hash_enc.get_feature_names()
hash_enc.get_params()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1123149328365526, 'fraction_numerical': 0.005833905284831846, 'mean_word_length': 5.457423235201013, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 70, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '295712', 'license': 'MIT', 'n_tokens_mistral': 6762, 'n_tokens_neox': 6590, 'n_words': 1657, 'path': 'Category Encoding.ipynb', 'repo_name': 'perryisbusy/Machine-Learning-Playground', 'size': '20398'}