<jupyter_start><jupyter_text>Encoding of categorical variables  Sources * [Encoding categorical variables](https://kiwidamien.github.io/encoding-categorical-variables.html)* [Benchmarking Categorical Encoders](https://towardsdatascience.com/benchmarking-categorical-encoders-9c322bd77ee8)* [Smarter Ways to Encode Categorical Data for Machine Learning](https://towardsdatascience.com/smarter-ways-to-encode-categorical-data-for-machine-learning-part-1-of-3-6dca2f71b159)* [Encoding Categorical Variables in Practice](https://medium.com/epfl-extension-school/encoding-categorical-variables-in-practice-a536907f2013)* [Types of Categorical Data Encoding Schemes](https://medium.com/analytics-vidhya/types-of-categorical-data-encoding-schemes-a5bbeb4ba02b)* [Guide to Encoding Categorical Values in Python ](https://pbpython.com/categorical-encoding.html)* [All about Categorical Variable Encoding](https://towardsdatascience.com/all-about-categorical-variable-encoding-305f3361fd02)* [Encoding categorical variables: one-hot and beyond](http://www.win-vector.com/blog/2017/04/encoding-categorical-variables-one-hot-and-beyond/)  Background Many machine learning algorithms are not able to use non-numeric data. While many features we might use, such as a person's age, or height, are numeric there are many that are not. Usually these features are represented by strings, and we need some way of transforming them to numbers before using scikit-learn's algorithms. The different ways of doing this are called encodings. **Note**:This notebook does not address the translation of text into numbers/vectors. This is the topic of __word embeddings__ where the aim is to generate vectors such that words with similar meaning are close to one another and that the arithmetic of vectors corresponds to ''arithmetic'' of meaning, e.g. embedding('queen') ~ embedding('king') - embedding('man') + embedding('woman').  Terminology **Levels**: A levels of a non-numeric feature are the number of distinct values. The examples listed above are all examples of levels. The number of levels can vary wildly: the number of races for a patient is typically four (asian, black, hispanic, and white), the number of states for the US is 51 (if we include DC separately), while the number of professions is in the thousands. **Ordinal**: If the levels are ordered, then we call the feature ordinal. For example, if a class grade such as "B+" or "A" is a non-numeric feature, but the letters are not just different, they are ordered (an "A" is better than a "B+", which is better than a "C-" etc). **Nominal**: If the levels are just different without an ordering, we call the feature categorical. For example, professions or car brands are categorical. If we use an encoding that maps levels to numbers, we introduce an ordering on the categories, which may not be desirable. Most of this article will be about encoding categorical variables.  Considerations when choosing encodings **Do you have many levels?** If so, using an encoding that has a level-per-feature is difficult for tree-based models. Trees separate on features that "split" the data into different classes effectively. If there are many levels, it is likely only a tiny fraction of the data belong to one level, so it will be hard for trees to "find" that feature to split on. Typically this isn't a problem for linear models. **Are there many examples of each level?** If there are only 5 doctors in your dataset, you probably are not going to know the doctor category very well (nor will it generalize). Some encoders deal with this gracefully, while others won't. You might consider making an explicit "other" category for levels, or grouping categories together. This is a problem for all models. **Could you have new categories at test time?** Some categorical variables can be completely specified at training time (e.g. the levels for race or blood type would be known even with zero training examples). Other categories, such as profession, are so broad that we probably learn the levels from the training data. Some encoders deal with levels that are only in the test set better than others. **Are the categories related?** Many encoding schemes treat two different levels as "equally different" from one another. If looking at color of a car, a typical encoding has no idea that "brick red" and "red" are more related than "red" and "yellow". One way of solving this problem is to cluster the categories into higher levels, and then encode that category as well. **Is it reversible?** Does it store a lookup table? Given the encoding of a feature, can you recover the original value? If so, we call the encoding reversible. Generally, reversible features also require a lot of storage if there are a lot of levels (to figure out how to go backward). I would generally see reversible as a negative if it requries storing a lookup table as well.  Example data<jupyter_code>import requests
import category_encoders as ce
import numpy as np
import pandas as pd

def encode_var(var, encoder, y=None):
    if y is None:
        encoder.fit(var)
    else:
        encoder.fit(var, y)
    new_var = encoder.transform(var)
    if isinstance(new_var, pd.DataFrame):
        new_var.insert(0, 'original', var)
        return new_var
    else:
        return pd.DataFrame({'original': var, 'encoder': new_var})
    
def print_res(res, rows_per_level=2):
    out = pd.DataFrame(columns=res.columns)
    for lvl in res.original.unique():
        out = out.append(res[res.original==lvl].head(rows_per_level))
    return out
download_data = False
if download_data:
    url = "http://mlr.cs.umass.edu/ml/machine-learning-databases/autos/imports-85.data"
    r = requests.get(url)
    with open('imports-85.data', 'wb') as f:
        f.write(r.content)
# Define the headers since the data does not have any
headers = ["symboling", "normalized_losses", "make", "fuel_type", "aspiration",
           "num_doors", "body_style", "drive_wheels", "engine_location",
           "wheel_base", "length", "width", "height", "curb_weight",
           "engine_type", "num_cylinders", "engine_size", "fuel_system",
           "bore", "stroke", "compression_ratio", "horsepower", "peak_rpm",
           "city_mpg", "highway_mpg", "price"]
# Read in the CSV file and convert "?" to NaN
df = pd.read_csv("imports-85.data",
                  header=None, names=headers, na_values="?" )
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 205 entries, 0 to 204
Data columns (total 26 columns):
symboling            205 non-null int64
normalized_losses    164 non-null float64
make                 205 non-null object
fuel_type            205 non-null object
aspiration           205 non-null object
num_doors            203 non-null object
body_style           205 non-null object
drive_wheels         205 non-null object
engine_location      205 non-null object
wheel_base           205 non-null float64
length               205 non-null float64
width                205 non-null float64
height               205 non-null float64
curb_weight          205 non-null int64
engine_type          205 non-null object
num_cylinders        205 non-null object
engine_size          205 non-null int64
fuel_system          205 non-null object
bore                 201 non-null float64
stroke               201 non-null float64
compression_ratio    205 non-null float64
horsepower           203 non-[...]<jupyter_text>Notation $$ \begin{align}N: & \text{Number of observations}\\N_i: & \text{Number of observations with level i}\\n: & \text{Number of levels}\\i: & \text{The i-th level}\\k: & \text{The k-th variable of the encoding}\\x_k(i): & \text{Value for the k-th variable of the encoding when the original variable has label $i$.}\\\bar{y}: & \text{The average value of the target}\\\bar{y}_i: & \text{The average value of the target for level i}\end{align}$$  Unsupervised Encoding Methods  Label Encoding **Description**Label encoding replaces the *n* labels with values from *0* to *n-1*, in lexicographical order.**Construction**$$x(i) = i$$**When to use**Never (only for ordinal variables, for which there is the `OrdinalEncoder` class.<jupyter_code>df["num_cylinders"].value_counts().sort_index()
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
res = encode_var(df["num_cylinders"], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Ordinal Encoding **Description**Same as label encoding, but values are ordered in the order of labels.**When to use**For ordinal variables, where the difference between all labels represents the same distance.<jupyter_code>df["num_cylinders"].value_counts(sort=False)
df["num_cylinders"] = df["num_cylinders"].astype('category').cat.reorder_categories(ordered=True, new_categories=['two', 'three', 'four', 'five', 'six', 'eight', 'twelve'])
df["num_cylinders"].value_counts(sort=False)
encoder = ce.ordinal.OrdinalEncoder()
res = encode_var(df[["num_cylinders"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Dummy/One Hot Encoding **Description**Creates a binary indicator for each level.**Construction**$$x_k(i) = \begin{cases} 1 & \text{if } k=i \\ 0 & \text{otherwise}\end{cases}$$**When to use**When the main interest is in differences in average values for each level and there are sufficient observations for each level.<jupyter_code>df['drive_wheels'].value_counts()
encoder = ce.one_hot.OneHotEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Binary Encoding **Description**Similar to One-Hot Encoding, but values of categories are stored as binary bitstrings. Each binary digit create one encoding column, i.e. if there are $n$ levels then there are $\log_2n$ features.**Construction**Each level is associated with its order, which is then translated into bitstrings. The encoding variables are the different valeus in the bitstring.**When to use**For categorical variables with many levels.<jupyter_code>encoder = ce.binary.BinaryEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>BaseN Encoding **Description**Base-N encoder encodes the categories into arrays of their base-N representation. A base of $1$ is equivalent to one-hot encoding (not really base-1, but useful), a base of $2$ is equivalent to binary encoding. A base of $n$ is equivalent to vanilla ordinal encoding.**Construction**Each level is associated with its order, which is then translated into bitstrings. The encoding variables are the different valeus in the bitstring.**When to use**For categorical variables with many levels.<jupyter_code>encoder = ce.basen.BaseNEncoder(base=2)
res = encode_var(df[["make"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Simple Encoder **Description**Compares each level to the reference level, with the intercept as the grand mean.**Construction**$$x_k(i) = \begin{cases}\frac{n-1}{n} & \text{if } k = i \\-\frac{1}{n} & \text{otherwise}\end{cases}$$**When to use**Same as dummy encoding, but when the interest is in deviations from the grand mean rather than deviations from the reference level.<jupyter_code>### Not available in Python
from simple_coding import SimpleEncoder
encoder = SimpleEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Sum encoding / Deviation Encoding / Effect Encoding **Description**Sum encoding compares each group effect to the grand mean, i.e. the mean of group means (which is not the overall mean). The encoding representation is the difference of indicator variables minus the indicator for one baseline group. Sum encoding is similar to One-Hot encoding, but the interpretation of effects is different (effect relative to the grand mean vs. effect relative to a baseline group).**Construction**$$x_k(i) = \begin{cases}-1 & \text{if } i = 1\\1 & \text{if } i+1 = k\\0 & \text{otherwise}\end{cases}$$**When to use**When the main interest is in differences in average values for each level compared to the grand mean and there are sufficient observations for each level.<jupyter_code>encoder = ce.sum_coding.SumEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>(Orthogonal) Polynomial Encoding **Description**Orthogonal polynomial coding is a form of trend analysis in that it is looking for the linear, quadratic and cubic trends in the categorical variable. Orthogonal polynomials are equations such that each is associated with a power of the variable.**Construction**Each encoding variable are the coefficients of the orthogonal polynomials of order $n-1$. See this [Post](https://stats.stackexchange.com/questions/105115/polynomial-contrasts-for-regression) for an explanation of the construction.**When to use**This type of coding system should be used only with an ordinal variable in which the levels are equally spaced.<jupyter_code>encoder = ce.polynomial.PolynomialEncoder()
res = encode_var(df[["num_cylinders"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Helmert Encoding **Description**Helmert coding compares each level of a categorical variable to the mean of the subsequent levels. The first contrast compares the mean of the dependent variable for the second level with the mean of the dependent variable for the first level. The second contrast compares the mean for the third level with the mean for the first two levels, etc.**Construction**$$x_k(i) = \begin{cases}-1 & \text{if } i < k\\k & \text{if } i = k\\0 & \text{otherwise}\end{cases}$$The representation for level k has -1 for each level before k and then k as value for the current level, starting at *k=2* up to *k=n*.**When to use**When levels of a categorical variable are ordered from lowest to highest or from smallest to largest.<jupyter_code>encoder = ce.helmert.HelmertEncoder()
res = encode_var(df[["body_style"]], encoder)
print_res(res)
df["body_style"].unique()<jupyter_output><empty_output><jupyter_text>Backward Difference Encoding **Description**Similar to Helmert encoding, but differences are taken with respect to the prior adjacent level (not all prior levels).**Construction**$$x_k(i) = \begin{cases}-\frac{n-k}{n} & \text{if } i \leq k\\\frac{n}{k} & \text{otherwise}\end{cases}$$**When to use**When levels of a categorical variable are ordered from lowest to highest or from smallest to largest, but the interest is in step-wise differences.<jupyter_code>encoder = ce.backward_difference.BackwardDifferenceEncoder()
res = encode_var(df[["body_style"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Frequency / Count Encoding **Description**Frequency encoding replaces the levels of a categorical variable with their absolute or relative frequency.**Construction**$$x(i) = \frac{N_i}{N}$$**When to use**When common or uncommon levels have similar influences.<jupyter_code>encoder = ce.count.CountEncoder(normalize=True)
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Hashing **Description**Hashing converts categorical variables to a higher dimensional space of integers, where the distance between two vectors of categorical variables in approximately maintained the transformed numerical dimensional space. **Construction**Any hash funtion from the `hashlib` package.**When to use**This method is advantageous when the cardinality of categorical is very high.<jupyter_code>encoder = ce.hashing.HashingEncoder()
res = encode_var(df[["drive_wheels"]], encoder)
print_res(res)<jupyter_output><empty_output><jupyter_text>Supervised Encoding Methods / Bayesian Encoders  Target / Mean / Impact / Likelihood Encoding **Description**Target encoding replaces each category with the average value of all ovservations for that category, mixed with a prior.For the case of *categorical target*: features are replaced with a blend of posterior probability of the target given particular categorical value and the prior probability of the target over all the training data.For the case of *continuous target*: features are replaced with a blend of the expected value of the target given particular categorical value and the expected value of the target over all the training data.**Construction**$$x(i) = \bar{y} (1-s) + s \bar{y}_i$$where $s$ is a smoothing parameter calculated as:$$s=\frac{1}{1+\exp{\left(-\frac{n-mdl}{a}\right)}}$$with 'mdl' standing for 'min data in leaf' and $a$ is a regularization parameter. The mdl defines a threshold where prior, i.e. the mean of the target, and target mean for a given category value have the same weight.**When to use**This encoding method brings out the relation between similar categories, but the connections are bounded within the categories and target itself. The advantages of the mean target encoding are that it does not affect the volume of the data and helps in faster learning. Target encoding is powerful in prediction tasks, but runs the risk of target leakage. The leakage can be controlled via regularization, data augmentation by adding noise to the encoding representation, or through double validation.<jupyter_code>encoder = ce.target_encoder.TargetEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)<jupyter_output><empty_output><jupyter_text>James-Stein Encoding **Description**Target encoding based on the James-Stein mean estimator, rather than the sample mean. The idea is to improve the estimation of the category's mean target by shrinking them towards the central average.**Construction**$$x(i) = \bar{y} B + (1-B) \bar{y}_i$$where $B$ is a shrinkage parameter. A common value is$$B=\frac{Var\left[y_k\right]}{Var\left[y_k\right]+Var\left[y\right]}$$but the value could also be set via cross-valdiation. The intuition behind the equation is that if the mean estiamte of a category is uncertain (high variance), then stronger shrinkage should be applied.**When to use**Same as for target encoder.<jupyter_code>encoder = ce.james_stein.JamesSteinEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)<jupyter_output><empty_output><jupyter_text>M-estimator Encoding **Description** A simplified version of the target encoder as it has only one hyerparameter.**Construction**$$x(i) = \frac{N_i + m \times \bar{y}}{\bar{y}_i + m}$$where $m$ is a smoothing parameter.**When to use**Same as for target encoder.<jupyter_code>encoder = ce.m_estimate.MEstimateEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)<jupyter_output><empty_output><jupyter_text>Leave One Out Encoding **Description**The encoding is calculated for each observation $j$ by calculating the average value of the target of all observations with the same target value as observation $j$ except $j$.**Construction**$$ x_k^{(j)} = \frac{a}{b}\frac{\sum_{i\neq j}(y_i(x_i==k)-y_j}{\sum_{i\neq j}x_i==k}$$**When to use**Same as for target encoder.<jupyter_code>encoder = ce.leave_one_out.LeaveOneOutEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)<jupyter_output><empty_output><jupyter_text>Catboost Encoding **Decription**Catboost is an improvement over the leave-one-out encoder. It is intended to overcome the target leackage problems.**Construction**Same as LOO encoder, but the sums only range up to the current observations:$$ x_k(i) = \frac{a}{b}\frac{\sum_{j\neq i, j\leq k}(y_j(x_j==k)-y_i}{\sum_{j\neq i, j\leq k}x_j==k}$$**When to use**<jupyter_code>encoder = ce.cat_boost.CatBoostEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['price'])
print_res(res)<jupyter_output><empty_output><jupyter_text>Weight of Evidence Encoder **Description**The encoding is a measure of the *strength* that seperates effect from no effect. The encoding can only be used for binary targets.**Construction**The encoding is calculated from a modified odds ratio, which is intended to prevent target leakage:$$ \begin{align}numerator =& \ \frac{N_i+a}{N_i\bar{y}_i + 2a} \\denominator =& \ \frac{N-N_i+a}{N\bar{y}-N_i\bar{y}_i  2a} \\x(i) =& \ \log\left(\frac{numerator}{denominator}\right)\end{align}$$**When to use**For binary targets<jupyter_code>encoder = ce.woe.WOEEncoder()
res = encode_var(df[["drive_wheels"]], encoder, df['fuel_type']=='gas')
print_res(res)<jupyter_output><empty_output><jupyter_text>Probability Ratio Encoding **Description**Probability Ratio Encoding is similar to Weight Of Evidence encoding, but the encoding is based on the ratio of probabilities of the positive to the negative class.**Construction**$$x(i) = \frac{y^+/N}{y_k^-/N}=\frac{y_k^+}{y_k^-}$$This is the same as WoE encoding with no regularization, i.e. $a=0$.**When to use**Same as weight of evidence encoding.<jupyter_code>encoder = ce.woe.WOEEncoder(regularization=0.)
res = encode_var(df[["drive_wheels"]], encoder, df['fuel_type']=='gas')
print_res(res)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10287615878298075, 'fraction_numerical': 0.00893748514380794, 'mean_word_length': 5.303865747677555, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 81, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '300670', 'license': 'Apache-2.0', 'n_tokens_mistral': 6156, 'n_tokens_neox': 5784, 'n_words': 2307, 'path': 'notes/06_encoders.ipynb', 'repo_name': 'pbr142/handson-ml', 'size': '21035'}