<jupyter_start><jupyter_text>Categorical EncodingsIn this segment we shall cover the various types of encodings used to process columns with categorical values. We have already seen applications of `LabelEncoder` and `OneHotEncoder`. The various types we are goint to use are:- Label Ecoding- One Hot Encoding- Count Encoding- Target Encoding and variations- Singular value decompositionThese methods will again be applied on the Kickstarter project data, and the model will be used to predict whether a Kickstarter campaign succeeds or not. We will compare the effect on accuracy that each of these technoques have on the baseline model, which is made using Label Encoding, with minimal hyperparameter tuning.In this segment we will use the `category_encoders package`, which can be installed using```bash$ conda install -c conda-forge category_encoders```  Baseline Model<jupyter_code>import os
print(f"Current working directory: {os.getcwd()}")
import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Read data
ks = pd.read_csv('./dataset/kickstarter_project/ks-projects-201801.csv',
                 parse_dates = ['launched', 'deadline'])

# Drop live projects
ks = ks.query('state != "live"')

# Add outcome column, "successful" = 1, others are 0
ks = ks.assign(outcome=(ks['state'] == "successful").astype(int))

# Timestamp features
ks = ks.assign(hour=ks.launched.dt.hour,
               day=ks.launched.dt.day,
               month=ks.launched.dt.month,
               year=ks.launched.dt.year)

# LABEL ENCODING
# categorical features to consider
cat_features = ['category', 'currency', 'country']
encoder = LabelEncoder()
encoded = ks[cat_features].apply(encoder.fit_transform)

# features to use with the model
data_cols = ['goal','hour', 'day', 'month', 'year', 'outcome']
baseline_data = ks[data_cols].join(encoded)
# Defining functions to help evaluate/test our encoded data

import lightgbm as lgb
from sklearn import metrics

def get_data_splits(dataframe, valid_fraction=0.1):
    valid_size = int(len(dataframe)*valid_fraction)
    # Split the dataframe
    train = dataframe[: -2*valid_size]
    valid = dataframe[-2*valid_size : -valid_size]
    test  = dataframe[-valid_size :]
    
    return train, valid, test

def train_model(train, valid):
    feature_cols = train.columns.drop('outcome')
    dtrain = lgb.Dataset(train[feature_cols], label=train['outcome'])
    dvalid = lgb.Dataset(valid[feature_cols], label=valid['outcome'])
    param = {'num_leaves': 64, 'objective': 'binary',
             'metric': 'auc', 'seed': 7}
    print("Training model!")
    bst = lgb.train(param, dtrain, num_boost_round=1000,
                   valid_sets = dvalid,
                   early_stopping_rounds=10, 
                   verbose_eval=False)
    valid_pred = bst.predict(valid[feature_cols])
    valid_score = metrics.roc_auc_score(valid['outcome'], valid_pred)
    print(f"Validation AUC scores: {valid_score:.4f}")
    return bst
train, valid, _ = get_data_splits(baseline_data)
bst = train_model(train, valid)
baseline_data.head()<jupyter_output><empty_output><jupyter_text>Count EncodingCount encoding replaces each value with the number of times it appears in the dataset.We'll use the `categorical-encodings` package to get this encoding. The encoder itself is available as `CountEncoder`. This encoder and other encoders in the package work like scikit-learn transformers with `.fit` and `.transform` methods.<jupyter_code>import category_encoders as ce
cat_features = ['category', 'currency', 'country']
count_enc = ce.CountEncoder()
count_encoded = count_enc.fit_transform(ks[cat_features])

data = baseline_data.join(count_encoded.add_suffix("_count"))
data.head()
# Training a model on the baseline data
train, valid, test = get_data_splits(data)
bst = train_model(train, valid)<jupyter_output>Training model!
Validation AUC scores: 0.7486<jupyter_text>Using Count encoding we have gained a slight increase in validation scores.   Target EncodingTarget encoding replaces a categorical value with the average value of the target for that value of the feature. *Any given categorical value is replaced with the average of outcome of all rows containing the given categorical value.* Since this method used targets to create new features, including the validation or test data in target encodings would be a form of *target leakage*. Thus we should learn the target encodings from training datasets *only*.The `category_encoders` package provides `TargetEncoder` function for target encoding. It can be implemented im a similar manner as `CountEncoder`<jupyter_code>import category_encoders as ce
cat_features = ['category', 'currency', 'country']

# Create the encoder
target_enc = ce.TargetEncoder(cols=cat_features)

train, valid, _ = get_data_splits(data)

# Fit the eccoder ising the categorical features and target
target_enc.fit(train[cat_features], train['outcome'])

# Transform the features, rename the columns with _target suffix, and join to dataframe
train = train.join(target_enc.transform(train[cat_features]).add_suffix('_target'))
valid = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_target'))

train.head()
bst = train_model(train, valid)<jupyter_output>Training model!
Validation AUC scores: 0.7491<jupyter_text>CatBoost EncodingSimilar to Target Encoding i.e. based on target probability for a given value. However for CatBoost the taget probability of each row is calculated from the rows above it rather than the whole training data (Target Encoding).<jupyter_code>cat_features = ['category', 'currency', 'country']
# Define the encoder
target_enc = ce.CatBoostEncoder(cols=cat_features)

train, valid, _ = get_data_splits(data)

# Learn the encodings from training dataset
target_enc.fit(train[cat_features], train['outcome'])

# Transform categorical features in the training and validation dataset
train = train.join(target_enc.transform(train[cat_features]).add_suffix('_cb'))
valid = valid.join(target_enc.transform(valid[cat_features]).add_suffix('_cb'))

train.head()
# Train model based on CatBoost Encoding
bst = train_model(train, valid)<jupyter_output>Training model!
Validation AUC scores: 0.7492<jupyter_text>CatBoost performs slightly better than Target Encoding.<jupyter_code>l = int(len(train) + len(valid) + len(test))
print(l)
len(data)
a = 100 * len(train)/len(data)
a
b = 100 * len(valid)/len(data)
b
c= 100 * len(test)/len(data)
c
a+b+c<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09701838405685154, 'fraction_numerical': 0.007106442144291673, 'mean_word_length': 5.309941520467836, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '283558', 'license': 'MIT', 'n_tokens_mistral': 1963, 'n_tokens_neox': 1870, 'n_words': 623, 'path': 'feature_engineering_2_categoricalEncodings.ipynb', 'repo_name': 'rakshitraj/kaggle', 'size': '6473'}