<jupyter_start><jupyter_text>Evaluating and exporting scikit-learn metrics in a Keras callback**Author:** [lukewood](https://lukewood.xyz)**Date created:** 10/07/2021**Last modified:** 10/07/2021**Description:** This example shows how to use Keras callbacks to evaluate and export non-TensorFlow based metrics.  Introduction[Keras callbacks](https://keras.io/api/callbacks/) allow for the execution of arbitrarycode at various stages of the Keras training process.  While Keras offers first-classsupport for metric evaluation, [Keras metrics](https://keras.io/api/metrics/) may onlyrely on TensorFlow code internally.While there are TensorFlow implementations of many metrics online, some metrics areimplemented using [NumPy](https://numpy.org/) or another Python-based numerical computation library.By performing metric evaluation inside of a Keras callback, we can leverage any existingmetric, and ultimately export the result to TensorBoard.  Jaccard score metricThis example makes use of a sklearn metric, `sklearn.metrics.jaccard_score()`, andwrites the result to TensorBoard using the `tf.summary` API.This template can be modified slightly to make it work with any existing sklearn metric.<jupyter_code>import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.layers as layers
from sklearn.metrics import jaccard_score
import numpy as np
import os


class JaccardScoreCallback(keras.callbacks.Callback):
    """Computes the Jaccard score and logs the results to TensorBoard."""

    def __init__(self, model, x_test, y_test, log_dir):
        self.model = model
        self.x_test = x_test
        self.y_test = y_test
        self.keras_metric = tf.keras.metrics.Mean("jaccard_score")
        self.epoch = 0
        self.summary_writer = tf.summary.create_file_writer(
            os.path.join(log_dir, model.name)
        )

    def on_epoch_end(self, batch, logs=None):
        self.epoch += 1
        self.keras_metric.reset_state()
        predictions = self.model.predict(self.x_test)
        jaccard_value = jaccard_score(
            np.argmax(predictions, axis=-1), self.y_test, average=None
        )
        self.keras_metric.update_state(jaccard_value)
        self._write_metric(
            self.keras_metric.name, self.keras_metric.result().numpy().astype(float)
        )

    def _write_metric(self, name, value):
        with self.summary_writer.as_default():
            tf.summary.scalar(
                name, value, step=self.epoch,
            )
            self.summary_writer.flush()<jupyter_output><empty_output><jupyter_text>Sample usageLet's test our `JaccardScoreCallback` class with a Keras model.<jupyter_code># Model / data parameters
num_classes = 10
input_shape = (28, 28, 1)

# The data, split between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
print("x_train shape:", x_train.shape)
print(x_train.shape[0], "train samples")
print(x_test.shape[0], "test samples")


# Convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

model = keras.Sequential(
    [
        keras.Input(shape=input_shape),
        layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
        layers.MaxPooling2D(pool_size=(2, 2)),
        layers.Flatten(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation="softmax"),
    ]
)

model.summary()

batch_size = 128
epochs = 15

model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
callbacks = [JaccardScoreCallback(model, x_test, np.argmax(y_test, axis=-1), "logs")]
model.fit(
    x_train,
    y_train,
    batch_size=batch_size,
    epochs=epochs,
    validation_split=0.1,
    callbacks=callbacks,
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10843085741581085, 'fraction_numerical': 0.017434917602101745, 'mean_word_length': 4.003584229390681, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '378326', 'license': 'Apache-2.0', 'n_tokens_mistral': 1374, 'n_tokens_neox': 1325, 'n_words': 337, 'path': 'examples/keras_recipes/ipynb/sklearn_metric_callbacks.ipynb', 'repo_name': 'wariua/keras-io-ko', 'size': '4187'}