<jupyter_start><jupyter_text>Classification with TensorFlow Decision Forests**Author:** [<NAME>](https://www.linkedin.com/in/khalid-salama-24403144/)**Date created:** 2022/01/15**Last modified:** 2022/01/15**Description:** Using TensorFlow Decision Forests for structured data classification.  Introduction[TensorFlow Decision Forests](https://www.tensorflow.org/decision_forests)is a collection of state-of-the-art algorithms of Decision Forest modelsthat are compatible with Keras APIs.The models include [Random Forests](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/RandomForestModel),[Gradient Boosted Trees](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/GradientBoostedTreesModel),and [CART](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/CartModel),and can be used for regression, classification, and ranking task.This example uses Gradient Boosted Trees model in binary classification ofstructured data, and covers the following scenarios:1. Build a decision forests model by specifying the input feature usage.2. Implement a custom *Binary Target encoder* as a [Keras Preprocessing layer](https://keras.io/api/layers/preprocessing_layers/)to encode the categorical features with respect to their target value co-occurrences,and then use the encoded features to build a decision forests model.3. Encode the categorical features as [embeddings](https://keras.io/api/layers/core_layers/embedding),train these embeddings in a simple linear model, and then use thetrained embeddings as inputs to build decision forests model.This example uses TensorFlow 2.7 or higher,as well as [TensorFlow Decision Forests](https://www.tensorflow.org/decision_forests),which you can install using the following command:```pythonpip install -U tensorflow_decision_forests```  Setup<jupyter_code>import math
import urllib
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow_decision_forests as tfdf<jupyter_output><empty_output><jupyter_text>Prepare the dataThis example uses the[United States Census Income Dataset](https://archive.ics.uci.edu/ml/datasets/Census-Income+%28KDD%29)provided by the [UC Irvine Machine Learning Repository](https://archive.ics.uci.edu/ml/index.php).The task is binary classification to determine whether a person makes over 50K a year.The dataset includes ~300K instances with 41 input features: 7 numerical featuresand 34 categorical features.First we load the data from the UCI Machine Learning Repository into a Pandas DataFrame.<jupyter_code>BASE_PATH = "https://archive.ics.uci.edu/ml/machine-learning-databases/census-income-mld/census-income"
CSV_HEADER = [
    l.decode("utf-8").split(":")[0].replace(" ", "_")
    for l in urllib.request.urlopen(f"{BASE_PATH}.names")
    if not l.startswith(b"|")
][2:]
CSV_HEADER.append("income_level")

train_data = pd.read_csv(f"{BASE_PATH}.data.gz", header=None, names=CSV_HEADER,)
test_data = pd.read_csv(f"{BASE_PATH}.test.gz", header=None, names=CSV_HEADER,)<jupyter_output><empty_output><jupyter_text>We convert the target column from string to integer.<jupyter_code>target_labels = [" - 50000.", " 50000+."]
train_data["income_level"] = train_data["income_level"].map(target_labels.index)
test_data["income_level"] = test_data["income_level"].map(target_labels.index)<jupyter_output><empty_output><jupyter_text>Now let's show the shapes of the training and test dataframes, and display some instances.<jupyter_code>print(f"Train data shape: {train_data.shape}")
print(f"Test data shape: {test_data.shape}")
print(train_data.head().T)<jupyter_output><empty_output><jupyter_text>Define dataset metadataHere, we define the metadata of the dataset that will be useful for encodingthe input features with respect to their types.<jupyter_code># Target column name.
TARGET_COLUMN_NAME = "income_level"
# Weight column name.
WEIGHT_COLUMN_NAME = "instance_weight"
# Numeric feature names.
NUMERIC_FEATURE_NAMES = [
    "age",
    "wage_per_hour",
    "capital_gains",
    "capital_losses",
    "dividends_from_stocks",
    "num_persons_worked_for_employer",
    "weeks_worked_in_year",
]
# Categorical features and their vocabulary lists.
CATEGORICAL_FEATURES_WITH_VOCABULARY = {
    feature_name: sorted(
        [str(value) for value in list(train_data[feature_name].unique())]
    )
    for feature_name in CSV_HEADER
    if feature_name
    not in list(NUMERIC_FEATURE_NAMES + [WEIGHT_COLUMN_NAME, TARGET_COLUMN_NAME])
}
# All features names.
FEATURE_NAMES = NUMERIC_FEATURE_NAMES + list(
    CATEGORICAL_FEATURES_WITH_VOCABULARY.keys()
)<jupyter_output><empty_output><jupyter_text>Configure hyperparametersYou can find all the parameters of the Gradient Boosted Tree model in the[documentation](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/GradientBoostedTreesModel)<jupyter_code>GROWING_STRATEGY = "BEST_FIRST_GLOBAL"
NUM_TREES = 250
MIN_EXAMPLES = 6
MAX_DEPTH = 5
SUBSAMPLE = 0.65
SAMPLING_METHOD = "RANDOM"
VALIDATION_RATIO = 0.1<jupyter_output><empty_output><jupyter_text>Implement a training and evaluation procedure<jupyter_code>def prepare_sample(features, target, weight):
    for feature_name in features:
        if feature_name in CATEGORICAL_FEATURES_WITH_VOCABULARY:
            if features[feature_name].dtype != tf.dtypes.string:
                # Convert categorical feature values to string.
                features[feature_name] = tf.strings.as_string(features[feature_name])
    return features, target, weight


def run_experiment(model, train_data, test_data, num_epochs=1, batch_size=None):

    train_dataset = tfdf.keras.pd_dataframe_to_tf_dataset(
        train_data, label=TARGET_COLUMN_NAME, weight=WEIGHT_COLUMN_NAME
    ).map(prepare_sample, num_parallel_calls=tf.data.AUTOTUNE)
    test_dataset = tfdf.keras.pd_dataframe_to_tf_dataset(
        test_data, label=TARGET_COLUMN_NAME, weight=WEIGHT_COLUMN_NAME
    ).map(prepare_sample, num_parallel_calls=tf.data.AUTOTUNE)

    model.fit(train_dataset, epochs=num_epochs, batch_size=batch_size)
    _, accuracy = model.evaluate(test_dataset, verbose=0)
    print(f"Test accuracy: {round(accuracy * 100, 2)}%")<jupyter_output><empty_output><jupyter_text>Create model inputs<jupyter_code>def create_model_inputs():
    inputs = {}
    for feature_name in FEATURE_NAMES:
        if feature_name in NUMERIC_FEATURE_NAMES:
            inputs[feature_name] = layers.Input(
                name=feature_name, shape=(), dtype=tf.float32
            )
        else:
            inputs[feature_name] = layers.Input(
                name=feature_name, shape=(), dtype=tf.string
            )
    return inputs<jupyter_output><empty_output><jupyter_text>Experiment 1: Decision Forests with raw features  Specify model input feature usagesYou can attach semantics to each feature to control how it is used by the model.If not specified, the semantics are inferred from the representation type.It is recommended to specify the [feature usages](https://www.tensorflow.org/decision_forests/api_docs/python/tfdf/keras/FeatureUsage)explicitly to avoid incorrect inferred semantics is incorrect.For example, a categorical value identifier (integer) will be be inferred as numerical,while it is semantically categorical.For numerical features, you can set the `discretized` parameters to the numberof buckets by which the numerical feature should be discretized.This makes the training faster but may lead to worse models.<jupyter_code>def specify_feature_usages(inputs):
    feature_usages = []

    for feature_name in inputs:
        if inputs[feature_name].dtype == tf.dtypes.float32:
            feature_usage = tfdf.keras.FeatureUsage(
                name=feature_name, semantic=tfdf.keras.FeatureSemantic.NUMERICAL
            )
        else:
            feature_usage = tfdf.keras.FeatureUsage(
                name=feature_name, semantic=tfdf.keras.FeatureSemantic.CATEGORICAL
            )

        feature_usages.append(feature_usage)
    return feature_usages<jupyter_output><empty_output><jupyter_text>Create a Gradient Boosted Trees modelWhen compiling a decision forests model, you may only provide extra evaluation metrics.The loss is specified in the model construction,and the optimizer is irrelevant to decision forests models.<jupyter_code>def create_gbt_model():
    gbt_model = tfdf.keras.GradientBoostedTreesModel(
        features=specify_feature_usages(create_model_inputs()),
        exclude_non_specified_features=True,
        growing_strategy=GROWING_STRATEGY,
        num_trees=NUM_TREES,
        max_depth=MAX_DEPTH,
        min_examples=MIN_EXAMPLES,
        subsample=SUBSAMPLE,
        validation_ratio=VALIDATION_RATIO,
        task=tfdf.keras.Task.CLASSIFICATION,
        loss="DEFAULT",
    )

    gbt_model.compile(metrics=[keras.metrics.BinaryAccuracy(name="accuracy")])
    return gbt_model<jupyter_output><empty_output><jupyter_text>Train and evaluate the modelNote that when training a Decision Forests model, only one epoch is needed toread the full dataset. Any extra steps will result in unnecessary slower training.Therefore, the default `num_epochs=1` is used in the `run_experiment` method.<jupyter_code>gbt_model = create_gbt_model()
run_experiment(gbt_model, train_data, test_data)<jupyter_output><empty_output><jupyter_text>Inspect the modelThe `model.summary()` method will display several types of information aboutyour decision trees model, model type, task, input features, and feature importance.<jupyter_code>print(gbt_model.summary())<jupyter_output><empty_output><jupyter_text>Experiment 2: Decision Forests with target encoding[Target encoding](https://dl.acm.org/doi/10.1145/507533.507538) is a common preprocessingtechnique for categorical features that convert them into numerical features.Using categorical features with high cardinality as-is may lead to overfitting.Target encoding aims to replace each categorical feature value with one or morenumerical values that represent its co-occurrence with the target labels.More precisely, given a categorical feature, the binary target encoder in this examplewill produce three new numerical features:1. `positive_frequency`: How many times each feature value occurred with a positive target label.2. `negative_frequency`: How many times each feature value occurred with a negative target label.3. `positive_probability`: The probability that the target label is positive,given the feature value, which is computed as`positive_frequency / (positive_frequency + negative_frequency)`.Note that target encoding is effective with models that cannot automaticallylearn dense representations to categorical features, such as decision forestsor kernel methods. If neural network models are used, its recommended toencode categorical features as embeddings.  Implement Binary Target EncoderFor simplicity, we assume that the inputs for the `adapt` and `call` methodsare in the expected data types and shapes, so no validation logic is added.<jupyter_code>class BinaryTargetEncoding(layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def adapt(self, data):
        # data is expected to be an integer numpy array to a Tensor shape [num_exmples, 2].
        # This contains feature values for a given feature in the dataset, and target values.

        # Convert the data to a tensor.
        data = tf.convert_to_tensor(data)
        # Separate the feature values and target values
        feature_values = tf.cast(data[:, 0], tf.dtypes.int64)
        target_values = tf.cast(data[:, 1], tf.dtypes.bool)

        print("Target encoding: Computing unique feature values...")
        # Get feature vocabulary.
        unique_feature_values = tf.sort(tf.unique(feature_values).y)

        print(
            "Target encoding: Computing frequencies for feature values with positive targets..."
        )
        # Filter the data where the target label is positive.
        positive_indices = tf.where(condition=target_values)
        postive_feature_values = tf.gather_nd(
            params=feature_values, indices=positive_indices
        )
        # Compute how many times each feature value occurred with a positive target label.
        positive_frequency = tf.math.unsorted_segment_sum(
            data=tf.ones(
                shape=(postive_feature_values.shape[0], 1), dtype=tf.dtypes.int32
            ),
            segment_ids=postive_feature_values,
            num_segments=unique_feature_values.shape[0],
        )

        print(
            "Target encoding: Computing frequencies for feature values with negative targets..."
        )
        # Filter the data where the target label is negative.
        negative_indices = tf.where(condition=tf.math.logical_not(target_values))
        negative_feature_values = tf.gather_nd(
            params=feature_values, indices=negative_indices
        )
        # Compute how many times each feature value occurred with a negative target label.
        negative_frequency = tf.math.unsorted_segment_sum(
            data=tf.ones(
                shape=(negative_feature_values.shape[0], 1), dtype=tf.dtypes.int32
            ),
            segment_ids=negative_feature_values,
            num_segments=unique_feature_values.shape[0],
        )

        print("Target encoding: Storing target encoding statistics...")
        self.positive_frequency_lookup = tf.constant(positive_frequency)
        self.negative_frequency_lookup = tf.constant(negative_frequency)

    def reset_state(self):
        self.positive_frequency_lookup = None
        self.negative_frequency_lookup = None

    def call(self, inputs):
        # inputs is expected to be an integer numpy array to a Tensor shape [num_exmples, 1].
        # This includes the feature values for a given feature in the dataset.

        # Raise an error if the target encoding statistics are not computed.
        if (
            self.positive_frequency_lookup == None
            or self.negative_frequency_lookup == None
        ):
            raise ValueError(
                f"You need to call the adapt method to compute target encoding statistics."
            )

        # Convert the inputs to a tensor.
        inputs = tf.convert_to_tensor(inputs)
        # Cast the inputs int64 a tensor.
        inputs = tf.cast(inputs, tf.dtypes.int64)
        # Lookup positive frequencies for the input feature values.
        positive_fequency = tf.cast(
            tf.gather_nd(self.positive_frequency_lookup, inputs),
            dtype=tf.dtypes.float32,
        )
        # Lookup negative frequencies for the input feature values.
        negative_fequency = tf.cast(
            tf.gather_nd(self.negative_frequency_lookup, inputs),
            dtype=tf.dtypes.float32,
        )
        # Compute positive probability for the input feature values.
        positive_probability = positive_fequency / (
            positive_fequency + negative_fequency
        )
        # Concatenate and return the looked-up statistics.
        return tf.concat(
            [positive_fequency, negative_fequency, positive_probability], axis=1
        )<jupyter_output><empty_output><jupyter_text>Let's test the binary target encoder<jupyter_code>data = tf.constant(
    [
        [0, 1],
        [2, 0],
        [0, 1],
        [1, 1],
        [1, 1],
        [2, 0],
        [1, 0],
        [0, 1],
        [2, 1],
        [1, 0],
        [0, 1],
        [2, 0],
        [0, 1],
        [1, 1],
        [1, 1],
        [2, 0],
        [1, 0],
        [0, 1],
        [2, 0],
    ]
)

binary_target_encoder = BinaryTargetEncoding()
binary_target_encoder.adapt(data)
print(binary_target_encoder([[0], [1], [2]]))<jupyter_output><empty_output><jupyter_text>Implement a feature encoding with target encoding<jupyter_code>def create_target_encoder():
    inputs = create_model_inputs()
    target_values = train_data[[TARGET_COLUMN_NAME]].to_numpy()
    encoded_features = []
    for feature_name in inputs:
        if feature_name in CATEGORICAL_FEATURES_WITH_VOCABULARY:
            vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[feature_name]
            # Create a lookup to convert string values to an integer indices.
            # Since we are not using a mask token nor expecting any out of vocabulary
            # (oov) token, we set mask_token to None and  num_oov_indices to 0.
            lookup = layers.StringLookup(
                vocabulary=vocabulary, mask_token=None, num_oov_indices=0
            )
            # Convert the string input values into integer indices.
            value_indices = lookup(inputs[feature_name])
            # Prepare the data to adapt the target encoding.
            print("### Adapting target encoding for:", feature_name)
            feature_values = train_data[[feature_name]].to_numpy().astype(str)
            feature_value_indices = lookup(feature_values)
            data = tf.concat([feature_value_indices, target_values], axis=1)
            feature_encoder = BinaryTargetEncoding()
            feature_encoder.adapt(data)
            # Convert the feature value indices to target encoding representations.
            encoded_feature = feature_encoder(tf.expand_dims(value_indices, -1))
        else:
            # Expand the dimensions of the numerical input feature and use it as-is.
            encoded_feature = tf.expand_dims(inputs[feature_name], -1)
        # Add the encoded feature to the list.
        encoded_features.append(encoded_feature)
    # Concatenate all the encoded features.
    encoded_features = tf.concat(encoded_features, axis=1)
    # Create and return a Keras model with encoded features as outputs.
    return keras.Model(inputs=inputs, outputs=encoded_features)<jupyter_output><empty_output><jupyter_text>Create a Gradient Boosted Trees model with a preprocessorIn this scenario, we use the target encoding as a preprocessor for the Gradient Boosted Tree model,and let the model infer semantics of the input features.<jupyter_code>def create_gbt_with_preprocessor(preprocessor):

    gbt_model = tfdf.keras.GradientBoostedTreesModel(
        preprocessing=preprocessor,
        growing_strategy=GROWING_STRATEGY,
        num_trees=NUM_TREES,
        max_depth=MAX_DEPTH,
        min_examples=MIN_EXAMPLES,
        subsample=SUBSAMPLE,
        validation_ratio=VALIDATION_RATIO,
        task=tfdf.keras.Task.CLASSIFICATION,
    )

    gbt_model.compile(metrics=[keras.metrics.BinaryAccuracy(name="accuracy")])

    return gbt_model<jupyter_output><empty_output><jupyter_text>Train and evaluate the model<jupyter_code>gbt_model = create_gbt_with_preprocessor(create_target_encoder())
run_experiment(gbt_model, train_data, test_data)<jupyter_output><empty_output><jupyter_text>Experiment 3: Decision Forests with trained embeddingsIn this scenario, we build an encoder model that codes the categoricalfeatures to embeddings, where the size of the embedding for a given categoricalfeature is the square root to the size of its vocabulary.We train these embeddings in a simple linear model through backpropagation.After the embedding encoder is trained, we used it as a preprocessor to theinput features of a Gradient Boosted Tree model.Note that the embeddings and a decision forest model cannot be trainedsynergically in one phase, since decision forest models do not train with backpropagation.Rather, embeddings has to be trained in an initial phase,and then used as static inputs to the decision forest model.  Implement feature encoding with embeddings<jupyter_code>def create_embedding_encoder():
    inputs = create_model_inputs()
    encoded_features = []
    for feature_name in inputs:
        if feature_name in CATEGORICAL_FEATURES_WITH_VOCABULARY:
            vocabulary = CATEGORICAL_FEATURES_WITH_VOCABULARY[feature_name]
            # Create a lookup to convert string values to an integer indices.
            # Since we are not using a mask token nor expecting any out of vocabulary
            # (oov) token, we set mask_token to None and  num_oov_indices to 0.
            lookup = layers.StringLookup(
                vocabulary=vocabulary, mask_token=None, num_oov_indices=0
            )
            # Convert the string input values into integer indices.
            value_index = lookup(inputs[feature_name])
            # Create an embedding layer with the specified dimensions
            vocabulary_size = len(vocabulary)
            embedding_size = int(math.sqrt(vocabulary_size))
            feature_encoder = layers.Embedding(
                input_dim=len(vocabulary), output_dim=embedding_size
            )
            # Convert the index values to embedding representations.
            encoded_feature = feature_encoder(value_index)
        else:
            # Expand the dimensions of the numerical input feature and use it as-is.
            encoded_feature = tf.expand_dims(inputs[feature_name], -1)
        # Add the encoded feature to the list.
        encoded_features.append(encoded_feature)
    # Concatenate all the encoded features.
    encoded_features = layers.concatenate(encoded_features, axis=1)
    # Create and return a Keras model with encoded features as outputs.
    return keras.Model(inputs=inputs, outputs=encoded_features)<jupyter_output><empty_output><jupyter_text>Build a linear model to train the embeddings<jupyter_code>def create_linear_model(encoder):
    inputs = create_model_inputs()
    embeddings = encoder(inputs)
    linear_output = layers.Dense(units=1, activation="sigmoid")(embeddings)

    linear_model = keras.Model(inputs=inputs, outputs=linear_output)
    linear_model.compile(
        optimizer=keras.optimizers.Adam(),
        loss=keras.losses.BinaryCrossentropy(),
        metrics=[keras.metrics.BinaryAccuracy("accuracy")],
    )
    return linear_model


embedding_encoder = create_embedding_encoder()
run_experiment(
    create_linear_model(embedding_encoder),
    train_data,
    test_data,
    num_epochs=3,
    batch_size=256,
)<jupyter_output><empty_output><jupyter_text>Train and evaluate a Gradient Boosted Tree model with embeddings<jupyter_code>gbt_model = create_gbt_with_preprocessor(embedding_encoder)
run_experiment(gbt_model, train_data, test_data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07773250347020105, 'fraction_numerical': 0.0081493753638114, 'mean_word_length': 3.891371003066141, 'pattern_counts': {'":': 1, '<': 82, '<?xml version=': 0, '>': 82, 'https://': 14, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '454176', 'license': 'Apache-2.0', 'n_tokens_mistral': 6545, 'n_tokens_neox': 6275, 'n_words': 1764, 'path': 'examples/structured_data/ipynb/classification_with_tfdf.ipynb', 'repo_name': 'Ahn815/keras-io', 'size': '22340'}