<jupyter_start><jupyter_text>Body Weight PredictorThis notebook is an adaptation of the [Tensorflow Time series forecasting tutorial](https://www.tensorflow.org/tutorials/structured_data/time_series) that trains a bunch of models on the daily-updated [emarock/my-squash-data dataset](https://github.com/emarock/my-squash-data) and uses them to make predictions of future body weight values.[![View source on Github](https://img.shields.io/badge/-View%20source%20on%20Github-blue?logo=github&labelColor=black)](https://github.com/emarock/my-squash-data/blob/main/weight_predictor.ipynb)  Initialization  Imports and Constants<jupyter_code>import os
import datetime

import IPython
import IPython.display

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import tensorflow as tf
from tensorflow import keras
BATCH_SIZE = 32
MODEL_SIZE = 8
MAX_EPOCHS = 400
TRAIN_PATIENCE=10
NORMALIZE=True<jupyter_output><empty_output><jupyter_text>Retrieve the Dataset<jupyter_code>%rm -f ${DATASET_FILE}

DATASET_FILE = keras.utils.get_file(
    origin='https://raw.githubusercontent.com/emarock/my-squash-data/main/data.csv',
    fname='data.csv')

raw_df = pd.read_csv(DATASET_FILE)<jupyter_output><empty_output><jupyter_text>Cleanup and Aggregate Let's peek at the data:<jupyter_code>raw_df.head(10)<jupyter_output><empty_output><jupyter_text>At a first sight it looks like a pretty sparse table with one row per day and a large number of columns. Missing values are *NaN* and some days are missing (e.g. 2019-08-04/05). Let's create an index for the data frame based on the *Date* column and reindex to add rows for the missing days:<jupyter_code>raw_df.index = pd.DatetimeIndex(raw_df.pop('Date'))
raw_df = raw_df.reindex(pd.date_range(raw_df.index[0], raw_df.index[-1]))
raw_df.head(10)<jupyter_output><empty_output><jupyter_text>Most of the columns have too few values to be of any use:<jupyter_code>raw_df.describe().transpose().sort_values('count', ascending=False)<jupyter_output><empty_output><jupyter_text>Let's aggregate strictly related columns:<jupyter_code>def aggregate_duration_and_intensity(data_frame, src_prefix, dst_prefix):
    src_dur_col = '%s Duration (min)' % src_prefix
    src_int_col = '%s Intensity (1-5)' % src_prefix
    dst_dur_col = '%s Duration (min)' % dst_prefix
    dst_int_col = '%s Intensity (1-5)' % dst_prefix

    # Here we're calculating the weighted average on columns with
    # plenty of NaN. We're basically doing:
    #
    #            dst_int * dst_dur + src_int * src_dur
    # dst_int = ---------------------------------------
    #                     dst_dur + src_dur
    #
    # dst_dur = dst_dur + src_dur
    
    final_dur = data_frame[dst_dur_col].add(
        data_frame[src_dur_col], fill_value=0)

    data_frame[dst_int_col] = data_frame[dst_int_col].mul(
        data_frame[dst_dur_col], fill_value=0).add(
            data_frame[src_int_col].mul(
                data_frame[src_dur_col], fill_value=0), fill_value=0).div(
                    final_dur, fill_value=0)

    data_frame[dst_dur_col] = final_dur
    data_frame.pop(src_dur_col)
    data_frame.pop(src_int_col)

opponent_level = ['Very Favoured', 'Favoured', 'Equally Favoured',
                  'Unfavoured', 'Very Unfavoured', 'Unpredictable']
game_types = ['Casual Playing', 'Practice Match', 'Tournament Match']

dst_col_suffixes = [
    'Games Duration (min)',
    'Games Intensity (1-5)',
    'Games Played',
    'Games Won'
]

agr_df = raw_df.copy()

for l in opponent_level:
    for c in dst_col_suffixes:
        dst_col = '%s %s' % (l, c)
        agr_df[dst_col] = np.nan
    for t in game_types:

        dur_col = '%s %s Duration (min)' % (l, t)
        int_col = '%s %s Intensity (1-5)' % (l, t)
        if dur_col in raw_df.columns and int_col in raw_df.columns:
            aggregate_duration_and_intensity(agr_df,
                                             '%s %s' % (l, t), '%s Games' % l)
        for c in ['Games Played', 'Games Won']:
            src_col = '%s %s %s' % (l, t, c)
            dst_col = '%s %s' % (l, c)
            if src_col in raw_df.columns:
                agr_df[dst_col] = agr_df[dst_col].add(
                    agr_df.pop(src_col), fill_value=0)

agr_df.describe().transpose().sort_values('count', ascending=False)<jupyter_output><empty_output><jupyter_text>Let's see how values evolve over time:<jupyter_code>plot_cols = ['Weight (kg)', 'Coached Session Duration (min)',
             'On Court Drills Duration (min)']
plot_features = agr_df[plot_cols]
_ = plot_features.plot(subplots=True)<jupyter_output><empty_output><jupyter_text>Let's clean up the table a little more, by getting rid of non-essential columns and replacing *NaN* values either by interpolation (for missing "Weight" values) or with 0 (elsewhere).<jupyter_code>df = agr_df.drop(
    columns=[c for c in agr_df.columns if 'Played' in c or 'Won' in c or 'Intensity' in c or 'Bike' in c])
    #columns=[c for c in agr_df.columns if 'Played' in c or 'Won' in c or 'Bike' in c])
df['Weight (kg)'] = df['Weight (kg)'].interpolate()
df = df.fillna(0)
df.head()
df.describe().transpose()
plot_cols = ['Weight (kg)', 'Coached Session Duration (min)',
             'On Court Drills Duration (min)']
plot_features = df[plot_cols]
_ = plot_features.plot(subplots=True)<jupyter_output><empty_output><jupyter_text>Since we are going to teach models to predict weight, we can assume there may be same weekly seasonality in play. We add two extra columns to signal that (following the [approach of the original tutorial](https://www.tensorflow.org/tutorials/structured_data/time_seriestime)).<jupyter_code>timestamp_d = df.index.map(datetime.datetime.timestamp) / (60 * 60 * 24)
df['Day of Week Sin'] = np.sin(timestamp_d * (2 * np.pi / 7))
df['Day of Week Cos'] = np.cos(timestamp_d * (2 * np.pi / 7))

plt.plot(np.array(df['Day of Week Sin'])[:7])
plt.plot(np.array(df['Day of Week Cos'])[:7])
plt.xlabel('Day')
plt.title('Day of Week signal')<jupyter_output><empty_output><jupyter_text>Split and Normalize(See https://www.tensorflow.org/tutorials/structured_data/time_seriessplit_the_data)<jupyter_code>column_indices = {name: i for i, name in enumerate(df.columns)}

n = len(df)
train_df = df[0:int(n*0.7)]
val_df = df[int(n*0.7):int(n*0.9)]
test_df = df[int(n*0.9):]

num_features = df.shape[1]
train_mean = train_df.mean()
train_std = train_df.std()

if NORMALIZE:
    train_df = (train_df - train_mean) / train_std
    val_df = (val_df - train_mean) / train_std
    test_df = (test_df - train_mean) / train_std
if NORMALIZE:
    df_std = (df - train_mean) / train_std
else:
    df_std = df.copy()

df_std = df_std.melt(var_name='Column',
                     value_name='Value')
plt.figure(figsize=(12, 6))
ax = sns.violinplot(x='Column', y='Value', data=df_std)
_ = ax.set_xticklabels(df.keys(), rotation=90)<jupyter_output><empty_output><jupyter_text>Data Windowing(See https://www.tensorflow.org/tutorials/structured_data/time_seriesdata_windowing)<jupyter_code>class WindowGenerator():
  def __init__(self, input_width, label_width, shift,
               train_df=train_df, val_df=val_df, test_df=test_df,
               label_columns=None):
    # Store the raw data.
    self.train_df = train_df
    self.val_df = val_df
    self.test_df = test_df

    # Work out the label column indices.
    self.label_columns = label_columns
    if label_columns is not None:
      self.label_columns_indices = {name: i for i, name in
                                    enumerate(label_columns)}
    self.column_indices = {name: i for i, name in
                           enumerate(train_df.columns)}

    # Work out the window parameters.
    self.input_width = input_width
    self.label_width = label_width
    self.shift = shift

    self.total_window_size = input_width + shift

    self.input_slice = slice(0, input_width)
    self.input_indices = np.arange(self.total_window_size)[self.input_slice]

    self.label_start = self.total_window_size - self.label_width
    self.labels_slice = slice(self.label_start, None)
    self.label_indices = np.arange(self.total_window_size)[self.labels_slice]

  def __repr__(self):
    return '\n'.join([
        f'Total window size: {self.total_window_size}',
        f'Input indices: {self.input_indices}',
        f'Label indices: {self.label_indices}',
        f'Label column name(s): {self.label_columns}'])
def split_window(self, features):
  inputs = features[:, self.input_slice, :]
  labels = features[:, self.labels_slice, :]
  if self.label_columns is not None:
    labels = tf.stack(
        [labels[:, :, self.column_indices[name]] for name in self.label_columns],
        axis=-1)

  # Slicing doesn't preserve static shape information, so set the shapes
  # manually. This way the `tf.data.Datasets` are easier to inspect.
  inputs.set_shape([None, self.input_width, None])
  labels.set_shape([None, self.label_width, None])

  return inputs, labels

WindowGenerator.split_window = split_window
def plot(self, model=None, plot_col='Weight (kg)', max_subplots=3):
  inputs, labels = self.example
  plt.figure(figsize=(12, 8))
  plot_col_index = self.column_indices[plot_col]
  max_n = min(max_subplots, len(inputs))
  for n in range(max_n):
    plt.subplot(max_n, 1, n+1)
    plt.ylabel(f'{plot_col}{" [normed]" if NORMALIZE else ""}')
    plt.plot(self.input_indices, inputs[-n-1, :, plot_col_index],
             label='Inputs', marker='.', zorder=-10)

    if self.label_columns:
      label_col_index = self.label_columns_indices.get(plot_col, None)
    else:
      label_col_index = plot_col_index

    if label_col_index is None:
      continue

    plt.scatter(self.label_indices, labels[-n-1, :, label_col_index],
                edgecolors='k', label='Labels', c='#2ca02c', s=64)
    if model is not None:
      predictions = model(inputs)
      plt.scatter(self.label_indices, predictions[-n-1, :, label_col_index],
                  marker='X', edgecolors='k', label='Predictions',
                  c='#ff7f0e', s=64)

    if n == 0:
      plt.legend()

  plt.xlabel('Day')

WindowGenerator.plot = plot
def make_dataset(self, data, shuffle=True):
  data = np.array(data, dtype=np.float32)
  ds = tf.keras.preprocessing.timeseries_dataset_from_array(
      data=data,
      targets=None,
      sequence_length=self.total_window_size,
      sequence_stride=1,
      shuffle=shuffle,
      batch_size=BATCH_SIZE)

  ds = ds.map(self.split_window)

  return ds

WindowGenerator.make_dataset = make_dataset
@property
def train(self):
  return self.make_dataset(self.train_df)

@property
def val(self):
  return self.make_dataset(self.val_df)

@property
def test(self):
  return self.make_dataset(self.test_df, shuffle=False)

@property
def example(self):
  """Get and cache an example batch of `inputs, labels` for plotting."""
  result = getattr(self, '_example', None)
  if result is None:
    # No example batch was found, so get the most recent one from the
    # `.test` dataset
    result = None
    for result in iter(self.test):
        pass
    # And cache it for next time
    self._example = result
  return result

WindowGenerator.train = train
WindowGenerator.val = val
WindowGenerator.test = test
WindowGenerator.example = example<jupyter_output><empty_output><jupyter_text>Models Creation, Training and Testing  Helper Functions Here's the function for training the models:<jupyter_code>def compile_and_fit(model, window, patience=TRAIN_PATIENCE):
  early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss',
                                                    patience=patience,
                                                    mode='min')

  model.compile(loss=tf.losses.MeanSquaredError(),
                optimizer=tf.optimizers.Adam(),
                metrics=[tf.metrics.MeanAbsoluteError()])

  history = model.fit(window.train, epochs=MAX_EPOCHS,
                      validation_data=window.val,
                      callbacks=[early_stopping])
  return history<jupyter_output><empty_output><jupyter_text>Being the test data frame unshaffled, running a model on it's last rows returns the prediction for the next day:<jupyter_code>def predict_next_weight(model):
    pred = np.array(model(np.array([test_df[-21:]]))).flatten()
    return pred[-1] * train_std['Weight (kg)'] + train_mean['Weight (kg)']<jupyter_output><empty_output><jupyter_text>Baseline Model(See https://www.tensorflow.org/tutorials/structured_data/time_seriesbaseline)<jupyter_code>single_step_window = WindowGenerator(
    input_width=1, label_width=1, shift=1,
    label_columns=['Weight (kg)'])
single_step_window
for example_inputs, example_labels in single_step_window.train.take(1):
  print(f'Inputs shape (batch, time, features): {example_inputs.shape}')
  print(f'Labels shape (batch, time, features): {example_labels.shape}')
class Baseline(tf.keras.Model):
  def __init__(self, label_index=None):
    super().__init__()
    self.label_index = label_index

  def call(self, inputs):
    if self.label_index is None:
      return inputs
    result = inputs[:, :, self.label_index]
    return result[:, :, tf.newaxis]
baseline = Baseline(label_index=column_indices['Weight (kg)'])

baseline.compile(loss=tf.losses.MeanSquaredError(),
                 metrics=[tf.metrics.MeanAbsoluteError()])

val_performance = {}
performance = {}
prediction = {}
val_performance['Baseline'] = baseline.evaluate(single_step_window.val)
performance['Baseline'] = baseline.evaluate(single_step_window.test, verbose=0)
prediction['Baseline'] = predict_next_weight(baseline)
wide_window = WindowGenerator(
    input_width=21, label_width=21, shift=1,
    label_columns=['Weight (kg)'])

wide_window
print('Input shape:', wide_window.example[0].shape)
print('Output shape:', baseline(wide_window.example[0]).shape)
wide_window.plot(baseline)<jupyter_output><empty_output><jupyter_text>Linear Model(See https://www.tensorflow.org/tutorials/structured_data/time_serieslinear_model)<jupyter_code>linear = tf.keras.Sequential([
    tf.keras.layers.Dense(units=1)
])
print('Input shape:', single_step_window.example[0].shape)
print('Output shape:', linear(single_step_window.example[0]).shape)
history = compile_and_fit(linear, single_step_window)

IPython.display.clear_output()
val_performance['Linear'] = linear.evaluate(single_step_window.val)
performance['Linear'] = linear.evaluate(single_step_window.test, verbose=0)
prediction['Linear'] = predict_next_weight(linear)
print('Input shape:', wide_window.example[0].shape)
print('Output shape:', baseline(wide_window.example[0]).shape)
wide_window.plot(linear, max_subplots=5)
plt.bar(x = range(len(train_df.columns)),
        height=linear.layers[0].kernel[:,0].numpy())
axis = plt.gca()
axis.set_xticks(range(len(train_df.columns)))
_ = axis.set_xticklabels(train_df.columns, rotation=90)<jupyter_output><empty_output><jupyter_text>Multi-layer Dense Network Model(See https://www.tensorflow.org/tutorials/structured_data/time_seriesdense)<jupyter_code>dense = tf.keras.Sequential([
    tf.keras.layers.Dense(units=MODEL_SIZE, activation='relu'),
    tf.keras.layers.Dense(units=MODEL_SIZE, activation='relu'),
    tf.keras.layers.Dense(units=1)
])

history = compile_and_fit(dense, single_step_window)

IPython.display.clear_output()
val_performance['Dense'] = dense.evaluate(single_step_window.val)
performance['Dense'] = dense.evaluate(single_step_window.test, verbose=0)
prediction['Dense'] = predict_next_weight(dense)
wide_window.plot(dense, max_subplots=5)<jupyter_output><empty_output><jupyter_text>Convolutional Network Model(See https://www.tensorflow.org/tutorials/structured_data/time_seriesconvolution_neural_network)<jupyter_code>CONV_WIDTH = 21
conv_window = WindowGenerator(
    input_width=CONV_WIDTH,
    label_width=1,
    shift=1,
    label_columns=['Weight (kg)'])

conv_window
conv_window.plot()
plt.title("Given 21 days as input, predict 1 day into the future.")
conv_model = tf.keras.Sequential([
    tf.keras.layers.Conv1D(filters=MODEL_SIZE,
                           kernel_size=(CONV_WIDTH,),
                           activation='relu'),
    tf.keras.layers.Dense(units=MODEL_SIZE, activation='relu'),
    tf.keras.layers.Dense(units=1),
])
print("Conv model on `conv_window`")
print('Input shape:', conv_window.example[0].shape)
print('Output shape:', conv_model(conv_window.example[0]).shape)
history = compile_and_fit(conv_model, conv_window)

IPython.display.clear_output()
val_performance['Conv'] = conv_model.evaluate(conv_window.val)
performance['Conv'] = conv_model.evaluate(conv_window.test, verbose=0)
prediction['Conv'] = predict_next_weight(conv_model)
print("Wide window")
print('Input shape:', wide_window.example[0].shape)
print('Labels shape:', wide_window.example[1].shape)
print('Output shape:', conv_model(wide_window.example[0]).shape)
LABEL_WIDTH = 24
INPUT_WIDTH = LABEL_WIDTH + (CONV_WIDTH - 1)
wide_conv_window = WindowGenerator(
    input_width=INPUT_WIDTH,
    label_width=LABEL_WIDTH,
    shift=1,
    label_columns=['Weight (kg)'])

wide_conv_window
print("Wide conv window")
print('Input shape:', wide_conv_window.example[0].shape)
print('Labels shape:', wide_conv_window.example[1].shape)
print('Output shape:', conv_model(wide_conv_window.example[0]).shape)
wide_conv_window.plot(conv_model)<jupyter_output><empty_output><jupyter_text>Recurrent Neural Network Model(See https://www.tensorflow.org/tutorials/structured_data/time_seriesrecurrent_neural_network)<jupyter_code>lstm_model = tf.keras.models.Sequential([
    # Shape [batch, time, features] => [batch, time, lstm_units]
    #tf.keras.layers.LSTM(MODEL_SIZE, return_sequences=True),
    tf.keras.layers.LSTM(16, return_sequences=True),
    # Shape => [batch, time, features]
    tf.keras.layers.Dense(units=1)
])
print('Input shape:', wide_window.example[0].shape)
print('Output shape:', lstm_model(wide_window.example[0]).shape)
history = compile_and_fit(lstm_model, wide_window, patience=5)

IPython.display.clear_output()
val_performance['LSTM'] = lstm_model.evaluate(wide_window.val)
performance['LSTM'] = lstm_model.evaluate(wide_window.test, verbose=0)
prediction['LSTM'] = predict_next_weight(lstm_model)
wide_window.plot(lstm_model)<jupyter_output><empty_output><jupyter_text>Performance Comparison<jupyter_code>x = np.arange(len(performance))
width = 0.3
metric_name = 'mean_absolute_error'
metric_index = lstm_model.metrics_names.index('mean_absolute_error')
val_mae = [v[metric_index] for v in val_performance.values()]
test_mae = [v[metric_index] for v in performance.values()]

plt.ylabel(f'mean_absolute_error [Weight (kg){", normalized]" if NORMALIZE else "]"}')
plt.bar(x - 0.17, val_mae, width, label='Validation')
plt.bar(x + 0.17, test_mae, width, label='Test')
plt.xticks(ticks=x, labels=performance.keys(),
           rotation=45)
_ = plt.legend()
for name, value in performance.items():
  print(f'{name:12s}: {value[1]:0.4f}')<jupyter_output>Baseline    : 0.6612
Linear      : 0.4876
Dense       : 0.5324
Conv        : 0.6267
LSTM        : 0.4413<jupyter_text>Tomorrow's Weight Prediction<jupyter_code>tomorrow = test_df.index[-1] + datetime.timedelta(days=1)
print(f'Predictions for {tomorrow.strftime("%b %d, %Y")}:')
for name, value in prediction.items():
  print(f'{name:12s}: {value:0.4f}')<jupyter_output>Predictions for Jun 10, 2021:
Baseline    : 72.8000
Linear      : 72.9583
Dense       : 73.0433
Conv        : 73.2472
LSTM        : 72.6693
{'fraction_non_alphanumeric': 0.13015106361113965, 'fraction_numerical': 0.012280341177679581, 'mean_word_length': 4.430524553571429, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 81, 'https://': 13, 'lorem ipsum': 0, 'www.': 9, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '615745', 'license': 'Apache-2.0', 'n_tokens_mistral': 6925, 'n_tokens_neox': 6621, 'n_words': 1324, 'path': 'weight_predictor.ipynb', 'repo_name': 'emarock/my-squash-data', 'size': '19462'}