<jupyter_start><jupyter_text>Discretisation plus EncodingWhat shall we do with the variable after discretisation? should we use the buckets as a numerical variable? or should we use the intervals as categorical variable?The answer is, you can do either.If you are building decision tree based algorithms and the output of the discretisation are integers (each integer referring to a bin), then you can use those directly, as decision trees will pick up non-linear relationships between the discretised variable and the target.If you are building linear models instead, the bins may not necessarily hold a linear relationship with the target. In this case, it may help improve model performance to treat the bins as categories and to one hot encoding, or target guided encodings like mean encoding, weight of evidence, or target guided ordinal encoding.We can easily do so by combining feature-engine's discretisers and encoders. In this demoWe will perform equal frequency discretisation followed by target guided orginal encoding using the titanic datasetIf instead you would like to do weight of evidence or mean target encoding, you need only replace the Feature-engine's encoder.<jupyter_code>import pandas as pd
import numpy as np

import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split

from feature_engine.discretisation import EqualFrequencyDiscretiser
from feature_engine.encoding import OrdinalEncoder
# load the the Titanic Dataset

data = pd.read_csv('../titanic.csv',
                   usecols=['age', 'fare', 'survived'])

data.head()
# Let's separate into train and test set

X_train, X_test, y_train, y_test = train_test_split(
    data[['age', 'fare']],
    data['survived'],
    test_size=0.3,
    random_state=0)

X_train.shape, X_test.shape<jupyter_output><empty_output><jupyter_text>The variables Age and Fare contain missing data, that I will fill by extracting a random sample of the variable.<jupyter_code>def impute_na(data, variable):

    df = data.copy()

    # random sampling
    df[variable + '_random'] = df[variable]

    # extract the random sample to fill the na
    random_sample = X_train[variable].dropna().sample(
        df[variable].isnull().sum(), random_state=0)

    # pandas needs to have the same index in order to merge datasets
    random_sample.index = df[df[variable].isnull()].index
    df.loc[df[variable].isnull(), variable + '_random'] = random_sample

    return df[variable + '_random']
# replace NA in both  train and test sets

X_train['age'] = impute_na(data, 'age')
X_test['age'] = impute_na(data, 'age')

X_train['fare'] = impute_na(data, 'fare')
X_test['fare'] = impute_na(data, 'fare')
# let's explore the distribution of age

X_train[['age', 'fare']].hist(bins=30, figsize=(8,4))
plt.show()<jupyter_output><empty_output><jupyter_text>Equal frequency discretisation with Feature-Engine<jupyter_code># set up the equal frequency discretiser
# to encode variables we need them returned as objects for feature-engine

disc = EqualFrequencyDiscretiser(
    q=10, variables=['age', 'fare'], return_object=True)

# find the intervals
disc.fit(X_train)

# transform train and text
train_t = disc.transform(X_train)
test_t = disc.transform(X_test)
train_t.dtypes
train_t.head()
# let's explore if the bins have a linear relationship
# with the target:

pd.concat([train_t, y_train], axis=1).groupby('age')['survived'].mean().plot()
plt.ylabel('mean of survived')
pd.concat([train_t, y_train], axis=1).groupby('fare')['survived'].mean().plot()
plt.ylabel('mean of survived')<jupyter_output><empty_output><jupyter_text>None of the variables show a monotonic relationship between the intervals of the discrete variable and the mean of survival. We can encode the intervals to return a monotonic relationship: Ordinal encoding with Feature-Engine<jupyter_code>enc = OrdinalEncoder(encoding_method = 'ordered')

enc.fit(train_t, y_train)

train_t = enc.transform(train_t)
test_t = enc.transform(test_t)
# in the map, we map bin to position

enc.encoder_dict_
pd.concat([train_t, y_train], axis=1).groupby('age')['survived'].mean().plot()
plt.ylabel('mean of survived')
pd.concat([train_t, y_train], axis=1).groupby('fare')['survived'].mean().plot()
plt.ylabel('mean of survived')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09951060358890701, 'fraction_numerical': 0.0032626427406199023, 'mean_word_length': 5.532724505327245, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '164797', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1292, 'n_tokens_neox': 1250, 'n_words': 449, 'path': '08.04-Discretisation-plus-Encoding.ipynb', 'repo_name': 'sri-spirited/feature-engineering-for-ml', 'size': '4291'}