<jupyter_start><jupyter_text>Categorical Encoding Demo and ExamplesThis is a Jupyter notebook for exploring the categorical-encoding library discussed in a [Feature Labs article] I wrote on the topic.To use the library, make sure you have the `categorical-encoding` and `featuretools` libraries installed. Encoder API<jupyter_code>import categorical_encoding as ce
import featuretools as ft

from featuretools.tests.testing_utils import make_ecommerce_entityset
es = make_ecommerce_entityset()
f1 = ft.Feature(es["log"]["product_id"])
f2 = ft.Feature(es["log"]["purchased"])
f3 = ft.Feature(es["log"]["value"])
f4 = ft.Feature(es["log"]["countrycode"])

features = [f1, f2, f3, f4]
ids = [0, 1, 2, 3, 4, 5]
feature_matrix = ft.calculate_feature_matrix(features, es,
                                             instance_ids=ids)
print(feature_matrix)<jupyter_output>product_id  purchased  value countrycode
id                                          
0    coke zero       True    0.0          US
1    coke zero       True    5.0          US
2    coke zero       True   10.0          US
3          car       True   15.0          US
4          car       True   20.0          US
5   toothpaste       True    0.0          AL<jupyter_text>Performing a train-test split is standard in machine learning pipelines. Here, I've just simulated an actual train-test split by randomly picking certain rows to be train or test data.<jupyter_code>train_data = feature_matrix.iloc[[0, 1, 4, 5]]
print(train_data)
test_data = feature_matrix.iloc[[2, 3]]
print(test_data)<jupyter_output>product_id  purchased  value countrycode
id                                         
2   coke zero       True   10.0          US
3         car       True   15.0          US<jupyter_text>Next up, we initialize and call the encoder on our data.<jupyter_code>enc = ce.Encoder(method='leave_one_out')

train_enc = enc.fit_transform(train_data, features, train_data['value'])

test_enc = enc.transform(test_data)
print(train_enc)
print(test_enc)<jupyter_output>PRODUCT_ID_leave_one_out  purchased  value  COUNTRYCODE_leave_one_out
id                                                                       
2                       2.50       True   10.0                   8.333333
3                       6.25       True   15.0                   8.333333<jupyter_text>Note how that the encoder only uses the training data to learn its encoding, and the test data is directly encoded using the learning mappings.Now, we typically would have to redo the entire categorical encoding process for the following feature matrix.<jupyter_code>fm2 = ft.calculate_feature_matrix(features, es, instance_ids=[6,7])
print(fm2)<jupyter_output>product_id  purchased  value countrycode
id                                          
6   toothpaste       True    1.0          AL
7   toothpaste       True    2.0          AL<jupyter_text>However, by integration with Featuretools, we can generate already encoded data.<jupyter_code>features_encoded = enc.get_features()

fm2_encoded = ft.calculate_feature_matrix(features_encoded, es, instance_ids=[6,7])
print(fm2_encoded)<jupyter_output>PRODUCT_ID_leave_one_out  purchased  value  COUNTRYCODE_leave_one_out
id                                                                       
6                       6.25       True    1.0                       6.25
7                       6.25       True    2.0                       6.25<jupyter_text>Encoding Methods ExamplesFor reference, here is our original encoder:<jupyter_code>feature_matrix<jupyter_output><empty_output><jupyter_text>Classic Encoders<jupyter_code># Creates a new column for each unique value. 
enc_one_hot = ce.Encoder(method='one_hot')
fm_enc_one_hot = enc_one_hot.fit_transform(feature_matrix, features)
fm_enc_one_hot
# Each unique string value is assigned a counting number specific to that value.
enc_ord = ce.Encoder(method='ordinal')
fm_enc_ord = enc_ord.fit_transform(feature_matrix, features)
fm_enc_ord
# The categories' values are first Ordinal encoded,
# the resulting integers are converted to binary,
# then the resulting digits are split into columns.
enc_bin = ce.Encoder(method='binary')
fm_enc_bin = enc_bin.fit_transform(feature_matrix, features)
fm_enc_bin
# use a hashing algorithm to map category values to corresponding columns
enc_hash = ce.Encoder(method='hashing')
fm_enc_hash = enc_hash.fit_transform(feature_matrix, features)
fm_enc_hash<jupyter_output><empty_output><jupyter_text>Bayesian Encoders<jupyter_code># replaces each specific category value with a weighted average of the dependent variable.
enc_targ = ce.Encoder(method='target')
fm_enc_targ = enc_targ.fit_transform(feature_matrix, features, feature_matrix['value'])
fm_enc_targ
# identical to target except leaves own row out when calculating average
enc_leave = ce.Encoder(method='leave_one_out')
fm_enc_leave = enc_leave.fit_transform(feature_matrix, features, feature_matrix['value'])
fm_enc_leave<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0706401766004415, 'fraction_numerical': 0.02187437286775035, 'mean_word_length': 2.946159936658749, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '411939', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1492, 'n_tokens_neox': 1449, 'n_words': 414, 'path': 'guides/notebooks/categorical-encoding-DEMO.ipynb', 'repo_name': 'isabella232/categorical_encoding', 'size': '4983'}