<jupyter_start><jupyter_text>Custom metricsIn an attempt to improve the [revamped](https://github.com/Gearlux/football-predictor/blob/master/02_revamped.ipynb) code,we are going to implement a custom metric which measures the profit. In order to do this, we need to supply the oddsto the keras metrics. I have not found a way to easily do this, but we can encode the odds and implement our own lossfunction which can handle the encoded labels. It turns out that these encoded labels can also be used to write a custom loss function, which optimizes the return of our betting. In addition, one could also think of a way not to bet on certain matches. This is also modeled with a fourth class and a new custom loss function.  Load the data and setup a keras environment<jupyter_code>import dataset
import keras

import pandas as pd
import numpy as np
import tensorflow as tf

from keras import regularizers
from keras import metrics
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adagrad
from keras.utils import np_utils
from keras_tqdm import TQDMNotebookCallback

from sklearn.preprocessing import StandardScaler, LabelEncoder

from keras import backend as K

np.set_printoptions(suppress=True)

book = dataset.Dataset('data/book.csv')
df = pd.DataFrame(book.processed_results)

def get_feables(df):
    features = df.drop(columns=['result','date'])
    encoder = LabelEncoder()
    labels = np_utils.to_categorical(-encoder.fit_transform(df.result.copy()) + 2)
    return features, labels

TRAINING_SET_FRACTION = 0.95
train_results_len = int(TRAINING_SET_FRACTION * df.shape[0])

features, labels = get_feables(df)
train_features = features[:train_results_len]
test_features = features[train_results_len:]
y_train = labels[:train_results_len]
y_test = labels[train_results_len:]

scaler = StandardScaler()
X_train = scaler.fit_transform(train_features.astype(float))
X_test = scaler.transform(test_features.astype(float))<jupyter_output>Using TensorFlow backend.<jupyter_text>UtilitiesTo make our live easier, we are going to implement a function which constructs our model with a specified- loss function- a list of metricsWe also want to have reproducible results. With the construction of each model we are setting the seed of some libraries.<jupyter_code>from numpy.random import seed
from tensorflow import set_random_seed
    
def construct_model(loss='categorical_crossentropy', metrics=['accuracy']):
    K.clear_session()

    seed(42)
    set_random_seed(42)

    model = Sequential()
    model.add(Dense(10, input_dim=X_train.shape[1],
                    kernel_regularizer=regularizers.l1(0.001)
                   ))
    model.add(Dense(3, 
                    kernel_regularizer=regularizers.l1(0.001),
                    activation = 'softmax'
                   ))
       
    model.compile(loss=loss, 
                  optimizer=Adagrad(0.1),
                  metrics=metrics
                 )

    return model<jupyter_output><empty_output><jupyter_text>In our [previous code](https://github.com/Gearlux/football-predictor/blob/master/02_revamped.ipynb), we have implemented a performance measure, relative to the odds of the bookmakers.You could also think of an absolute metric which does not use the bookmakers odds for comparison,but use the absolute probability of the model.<jupyter_code>def performance(th=0.05, features=test_features, name=None):
    def perf_metric(y_pred, y_true):
        odds = features[['odds-home','odds-draw','odds-away']]
        selection = (y_pred > (1./odds + th))
        profit = ( selection * (odds * y_true - 1)).sum()
        count = selection.sum()
        if name:
            profit.name = 'Profit ' + name
            count.name = 'Count ' + name
        return profit, count
    return perf_metric

def absolute_performance(th=0.9, features=test_features, name=None):
    def abs_perf_metric(y_pred, y_true):
        odds = features[['odds-home','odds-draw','odds-away']]
        selection = (y_pred > th)
        profit = ( selection * (odds * y_true - 1)).sum()
        count = selection.sum(axis=0)
        count = pd.Series(count, index=profit.index)
        if name:
            profit.name = '|Profit| ' + name
            count.name = '|Count| ' + name
        return profit, count
    return abs_perf_metric<jupyter_output><empty_output><jupyter_text>Original results<jupyter_code>import shutil
import os
if os.path.exists('logs'):
    shutil.rmtree('logs')
model = construct_model()
_ = model.fit(X_train, y_train,
      epochs=200,
      batch_size=500, verbose=0,
      validation_data = [X_test, y_test],
      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/original', write_graph=True), 
                 TQDMNotebookCallback(show_inner=False)]
 )
original = pd.concat([pd.concat(performance(0.05,name='original')(model.predict(X_test), y_test),axis=1),
pd.concat(absolute_performance(0.9,name='original')(model.predict(X_test), y_test),axis=1)], axis=1)
display(original)<jupyter_output><empty_output><jupyter_text>Custom loss functionThis custom loss function does not use the categorical encoded labels, but labels to also include the odds associated with the winning label.We will call the labels with the encoded winning odds as `b_train` or `b_test`.<jupyter_code>odds_train = train_features[['odds-home','odds-draw','odds-away']]
odds_test = test_features[['odds-home','odds-draw','odds-away']]

b_train = odds_train * (2 * y_train - 1)
b_test = odds_test * (2 * y_test - 1)<jupyter_output><empty_output><jupyter_text>The new labels new look like<jupyter_code>b_train.head()<jupyter_output><empty_output><jupyter_text>So the winning label has a positive odd and the losing labels have negative odds.Our loss function should be capable of dealing with our now labels.<jupyter_code>_EPSILON = 10e-8

def cat_loss(b_true, y_pred):
    prob_true = K.clip(b_true, 0., 1.)
    prob = K.clip(y_pred, _EPSILON, 1. - _EPSILON)
    res2 = K.sum(prob_true * -K.log(prob), axis=-1)
    return res2<jupyter_output><empty_output><jupyter_text>Let's create a new model which optimizes our new loss function.<jupyter_code>if os.path.exists('logs/cat_loss'):
    shutil.rmtree('logs/cat_loss')
    
model = construct_model(cat_loss)
_ = model.fit(X_train, b_train,
      epochs=200,
      batch_size=500, verbose=0,
      validation_data = [X_test, b_test],
      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/cat_loss', write_graph=True), 
                 TQDMNotebookCallback(show_inner=False)]
 )<jupyter_output><empty_output><jupyter_text>We also need to modify our performance functions.<jupyter_code>def b_performance(th=0.05, name=None):
    def perf_metric(y_pred, y_true):
        odds = np.abs(y_true)
        selection = (y_pred > (1./odds + th))
        profit = ( selection * (odds * np.clip(y_true, 0, 1) - 1)).sum(axis=0)
        count = selection.sum(axis=0)
        profit = pd.Series(profit)
        profit.index= ['home','draw','away']
        count = pd.Series(count, index=profit.index)
        if name:
            profit.name = 'Profit ' + name
            count.name = 'Count ' + name
        return profit, count
    return perf_metric

def b_absolute_performance(th=0.9, name=None):
    def abs_perf_metric(y_pred, y_true):
        odds = np.abs(y_true)

        selection = (y_pred > th)
        profit = ( selection * (odds * np.clip(y_true, 0, 1) - 1)).sum(axis=0)
        count = selection.sum(axis=0)
        profit = pd.Series(profit)
        profit.index= ['home','draw','away']
        count = pd.Series(count, index=profit.index)
        if name:
            profit.name = '|Profit| ' + name
            count.name = '|Count| ' + name
        return profit, count
    return abs_perf_metric
b = pd.concat([pd.concat(b_performance(0.05,name='b')(model.predict(X_test), b_test.values),axis=1),
pd.concat(b_absolute_performance(0.9,name='b')(model.predict(X_test), b_test.values),axis=1)], axis=1)
display(b)<jupyter_output><empty_output><jupyter_text>If we have implemented the `cat_loss` function correctly, we should have similar results as the original profit.<jupyter_code>display(original)<jupyter_output><empty_output><jupyter_text>Success! Because we have setup the seeds of both our libraries to the answer to the universe,we obtain exaclty the same result. Better metricsDon't forget the reason why we want to implement something similar.The reason why we want to implement this new loss function, is to define a metric which can shows us theprofit.In the previous section, we could only monitor the profit at the end of fitting process.It makes more sense to monitor the profit during the learning process.We implemented two performance metrics in the previous section.Likewise, we will also implement two metrics who will each implement both the profit and the margin which can be monitored in tensorboard.<jupyter_code>def bet_metric(th=0.05):
    def profit(y_true, y_pred):
        true_odds = K.abs(y_true)
        selection = K.cast((y_pred > (1./true_odds + th)),'float32')
        odds_true = K.clip(y_true, 0., np.inf) - 1
        return K.sum(K.sum( selection * odds_true))
    def margin(y_true, y_pred):
        true_odds = K.abs(y_true)
        selection = K.cast((y_pred > (1./true_odds + th)),'float32')
        odds_true = K.clip(y_true, 0., np.inf) - 1
        return K.sum(K.sum( selection * odds_true)) / K.sum(K.sum(selection))
    return profit, margin

def abs_bet_metric(th=0.9):
    def abs_profit(y_true, y_pred):
        selection = K.cast((y_pred > th),'float32')
        odds_true = K.clip(y_true, 0., np.inf) - 1
        return K.sum(K.sum( selection * odds_true))
    def abs_margin(y_true, y_pred):
        selection = K.cast((y_pred > th),'float32')
        odds_true = K.clip(y_true, 0., np.inf) - 1
        return K.sum(K.sum( selection * odds_true)) / K.sum(K.sum(selection))
    return abs_profit, abs_margin
if os.path.exists('logs/bet_metric'):
    shutil.rmtree('logs/bet_metric')
    
model = construct_model(cat_loss, metrics=['accuracy'] + list(bet_metric(0.05)) + list(abs_bet_metric(0.9)))
_ = model.fit(X_train, b_train,
      epochs=200,
      batch_size=500, verbose=0,
      validation_data = [X_test, b_test],
      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/bet_metric', write_graph=True), 
                 TQDMNotebookCallback(show_inner=False)]
 )
b = pd.concat([pd.concat(b_performance(0.05,name='b')(model.predict(X_test), b_test.values),axis=1),
pd.concat(b_absolute_performance(0.9,name='b')(model.predict(X_test), b_test.values),axis=1)], axis=1)
display(b)<jupyter_output><empty_output><jupyter_text>These are the same results as in the previous run, and surprise, surprise !In tensorboard, we can find the same end result for the profits and derived margins.Not only do we have the metric for our training set, but it's also available for the validation set.                  But the perfomance and the metrics, are similar functions. Can we share the code ?<jupyter_code>import tensorflow as tf

def profit_(y_true, y_pred, threshold):
    selection = K.cast((y_pred > threshold),'float32')
    odds_true = K.clip(y_true, 0., np.inf) - 1
    return K.sum( selection * odds_true, axis=0), K.sum(selection, axis=0)

def bet_metric(th=0.05):
    def profit(y_true, y_pred):
        true_odds = K.abs(y_true)
        condition = (1./true_odds + th)
        return K.sum(profit_(y_true, y_pred, condition)[0])
    def margin(y_true, y_pred):
        true_odds = K.abs(y_true)
        condition = (1./true_odds + th)
        pr = profit_(y_true, y_pred, condition)
        return K.sum(pr[0]) / K.sum(pr[1])
    return profit, margin

def abs_bet_metric(th=0.9):
    def abs_profit(y_true, y_pred):
        return K.sum(profit_(y_true, y_pred, th)[0])
    def abs_margin(y_true, y_pred):
        pr = profit_(y_true, y_pred, th)
        return K.sum(pr[0]) / K.sum(pr[1])
    return abs_profit, abs_margin

def b_performance(th=0.05, name=None):
    def perf_metric(y_pred, y_true):
        ypred = tf.convert_to_tensor(y_pred, dtype=tf.float32)
        ytrue = tf.convert_to_tensor(y_true, dtype=tf.float32)
        true_odds = K.abs(ytrue)
        condition = (1./true_odds + th)
        pr, co = profit_(ytrue, ypred, condition)
        idx = ['home','draw','away']
        profit = pd.Series(K.eval(pr), index=idx)
        count = pd.Series(K.eval(co), index=idx)
        if name:
            profit.name = 'Profit ' + name
            count.name = 'Count ' + name
        return profit, count
    return perf_metric

def b_absolute_performance(th=0.9, name=None):
    def abs_perf_metric(y_pred, y_true):
        ypred = tf.convert_to_tensor(y_pred, dtype=tf.float32)
        ytrue = tf.convert_to_tensor(y_true, dtype=tf.float32)
        pr, co = profit_(ytrue, ypred, th)
        idx = ['home','draw','away']
        profit = pd.Series(K.eval(pr), index=idx)
        count = pd.Series(K.eval(co), index=idx)
        if name:
            profit.name = '|Profit| ' + name
            count.name = '|Count| ' + name
        return profit, count
    return abs_perf_metric
if os.path.exists('logs/keras_metric'):
    try:
        shutil.rmtree('logs/keras_metric')
    except OSError:
        pass
    
model = construct_model(cat_loss, metrics=['accuracy'] + list(bet_metric(0.05)) + list(abs_bet_metric(0.9)))
_ = model.fit(X_train, b_train,
      epochs=200,
      batch_size=500, verbose=0,
      validation_data = [X_test, b_test],
      callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/keras_metric', write_graph=True), 
                 TQDMNotebookCallback(show_inner=False)]
 )
cm = pd.concat([pd.concat(b_performance(0.05,name='cm')(model.predict(X_test), b_test.values),axis=1),
pd.concat(b_absolute_performance(0.9,name='cm')(model.predict(X_test), b_test.values),axis=1)], axis=1)
display(cm)<jupyter_output><empty_output><jupyter_text>IntermissionWith our `cat_loss`, `bet_metric` and `abs_bet_metric` we now have tools to better monitor<jupyter_code>def clean_folds():
    for i in range(100):
        if os.path.exists('logs/fold_%02d' % i):
            try:
                shutil.rmtree('logs/fold_%02d' % i)
            except (PermissionError, OSError):
                pass
clean_folds()
from sklearn.model_selection import KFold
kfold = KFold(n_splits=20, random_state=42)

def cross_validate(epochs=200, loss=cat_loss):
    results = []
    models = []

    for i, (trainidx, valididx) in enumerate(kfold.split(df)):
        train = df.iloc[trainidx]
        test = df.iloc[valididx]
        train_features = train.drop(columns=['result'])
        train_labels = train.result.copy()
        test_features = test.drop(columns=['result'])
        test_labels = test.result.copy()
        scaler = StandardScaler()
        X_train = scaler.fit_transform(train_features.astype(float))
        X_test = scaler.transform(test_features.astype(float))
        encoder = LabelEncoder()
        Y_train = -encoder.fit_transform(train_labels) +2
        Y_test = -encoder.transform(test_labels) +2
        y_train = np_utils.to_categorical(Y_train)
        y_test = np_utils.to_categorical(Y_test)    
        odds_train = train_features[['odds-home','odds-draw','odds-away']]
        odds_test = test_features[['odds-home','odds-draw','odds-away']]
        c_train = odds_train * (2 * y_train - 1)
        c_test = odds_test * (2 * y_test - 1)

        model = construct_model(loss, metrics=['accuracy'] + list(bet_metric(0.05)) + list(abs_bet_metric(0.9)))

        _ = model.fit(X_train, c_train,
              epochs=epochs,
              batch_size=500, verbose=0,
              validation_data = [X_test, c_test],
              callbacks=[keras.callbacks.TensorBoard(log_dir='./logs/fold_%02d' % i, write_graph=True), 
                         TQDMNotebookCallback(show_inner=False)]
         )    

        cm = pd.concat([pd.concat(b_performance(0.05,name='cm')(model.predict(X_test), c_test.values),axis=1),
        pd.concat(b_absolute_performance(0.9,name='cm')(model.predict(X_test), c_test.values),axis=1)], axis=1)
        results.append(cm)
        models.append(model)

        display(cm)
        
    return results, models
results, models = cross_validate()<jupyter_output><empty_output><jupyter_text>Examine the profit of our folded model.<jupyter_code>result = pd.concat({n: df for n, df in enumerate(results)},axis=0)
result.sum(level=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11979831115971083, 'fraction_numerical': 0.014215418261344996, 'mean_word_length': 3.5740483467629898, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 65, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '208467', 'license': 'MIT', 'n_tokens_mistral': 5565, 'n_tokens_neox': 5366, 'n_words': 1251, 'path': '03_custom_metrics.ipynb', 'repo_name': 'Gearlux/football-predictor', 'size': '16461'}