<jupyter_start><jupyter_text>Deep learning models for age prediction on EEG data This notebook uses deep learning methods to predict the age of infants using EEG data. The EEG data is preprocessed as shown in the notebook 'Deep learning EEG_dataset preprocessing raw'.<jupyter_code>import sys, os, fnmatch, csv
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

sys.path.insert(0, os.path.dirname(os.getcwd()))

from config import PATH_DATA_PROCESSED_DL_REDUCED, PATH_MODELS<jupyter_output><empty_output><jupyter_text>Load preprocessed data (reduced) The data can be found in 'PATH_DATA_PROCESSED_DL_REDUCED'. This is a single folder with all the data and metadata. The EEG data is in the .zarr files and the metadata is in .csv files. The .zarr files are divided in chunks of 1-second epochs (average of 10 original EEG epochs) from the same subject and the metadata contains the information like the subject's identification code and age. The notebook "Deep learning EEG_DL dataset_reduction.ipynb" takes care of reducing the original processed data set to the reduced size.Generator loads all the data into memory. The generators generate averaged epochs on the fly. The data is split in train/validation/test and no subject can be found in more than one of these splits. Originally, we used the original EEG epochs and averaged 30 of them into a new EEG epoch in the generator object. This had two disadvantages: (1) many files had to be openened and closed during the training process, and (2) the data set was too large to fit into memory. Therefore, we decided to randomly create chunks of 10 EEG epochs and average those for each subject/age group. This reduced the data set from ±145GB to ±14.5 GB. We now use these already averaged EEG epochs as "original" epcohs, and average those another 3-5 times to reduce noise. We have also experimented with averaging all EEG epochs of a subject at a specific age into a single EEG epoch, but performance was lower, most likely because this reduced the size of the data set a lot.<jupyter_code>%%time

# Load all the metadata

from sklearn.model_selection import train_test_split

# Step 1: Get all the files in the output folder
file_names = os.listdir(PATH_DATA_PROCESSED_DL_REDUCED)

# Step 2: Get the full paths of the files (without extensions)
files = [os.path.splitext(os.path.join(PATH_DATA_PROCESSED_DL_REDUCED, file_name))[0] for file_name in fnmatch.filter(file_names, "*.zarr")]

# Step 3: Load all the metadata
frames = []

for idx, feature_file in enumerate(files):
    df_metadata = pd.read_csv(feature_file + ".csv")
    frames.append(df_metadata)

df_metadata = pd.concat(frames) 

# Step 4: Add missing age information based on the age group the subject is in
df_metadata['age_months'].fillna(df_metadata['age_group'], inplace=True)
df_metadata['age_days'].fillna(df_metadata['age_group']*30, inplace=True)
df_metadata['age_years'].fillna(df_metadata['age_group']/12, inplace=True)

# Step 5: List all the unique subject IDs
subject_ids = sorted(list(set(df_metadata["code"].tolist())))
from sklearn.model_selection import train_test_split

IDs_train, IDs_temp = train_test_split(subject_ids, test_size=0.3, random_state=42)
IDs_test, IDs_val = train_test_split(IDs_temp, test_size=0.5, random_state=42)
df_metadata
from dataset_generator_reduced import DataGeneratorReduced
# # Import libraries
# from tensorflow.keras.utils import Sequence
# import numpy as np
# import zarr
# import os

# class DataGeneratorReduced(Sequence):
#     """Generates data for loading (preprocessed) EEG timeseries data.
#     Create batches for training or prediction from given folders and filenames.

#     """
#     def __init__(self,
#                  list_IDs,
#                  BASE_PATH,
#                  metadata,
#                  gaussian_noise=0.0,
#                  n_average = 3,
#                  batch_size=10,
#                  iter_per_epoch = 1,
#                  n_timepoints = 501,
#                  n_channels=30,
#                  shuffle=True,
#                  warnings=False):
#         """Initialization

#         Args:
#         --------
#         list_IDs:
#             list of all filename/label ids to use in the generator
#         metadata:
#             DataFrame containing all the metadata.
#         n_average: int
#             Number of EEG/time series epochs to average.
#         batch_size:
#             batch size at each iteration
#         iter_per_epoch: int
#             Number of iterations over all data points within one epoch.
#         n_timepoints: int
#             Timepoint dimension of data.
#         n_channels:
#             number of input channels
#         shuffle:
#             True to shuffle label indexes after every epoch
#         """
#         self.list_IDs = list_IDs
#         self.BASE_PATH = BASE_PATH
#         self.metadata = metadata
#         self.metadata_temp = None
#         self.gaussian_noise = gaussian_noise
#         self.n_average = n_average
#         self.batch_size = batch_size
#         self.iter_per_epoch = iter_per_epoch
#         self.n_timepoints = n_timepoints
#         self.n_channels = n_channels
#         self.shuffle = shuffle
#         self.warnings = warnings
#         self.on_epoch_end()

#         # Store all data in here
#         self.X_data_all = []
#         self.y_data_all = []
#         self.load_all_data()


#     def __len__(self):
#         """Denotes the number of batches per epoch

#         return: number of batches per epoch
#         """
#         return int(np.floor(len(self.metadata_temp) / self.batch_size))


#     def __getitem__(self, index):
#         """Generate one batch of data

#         Args:
#         --------
#         index: int
#             index of the batch

#         return: X and y when fitting. X only when predicting
#         """
        
#         # Generate indexes of the batch
#         indexes = self.indexes[index * self.batch_size:((index + 1) * self.batch_size)]

#         # Generate data
#         X, y = self.generate_data(indexes)

#         return X, y

#     def load_all_data(self):
#         """ Loads all data into memory. """
#         for i, metadata_file in self.metadata_temp.iterrows():
#             filename = os.path.join(self.BASE_PATH, metadata_file['cnt_file'] + '.zarr')
            
#             X_data = np.zeros((0, self.n_channels, self.n_timepoints))

#             data_signal = self.load_signal(filename)

#             if (len(data_signal) == 0) and self.warnings:
#                 print(f"EMPTY SIGNAL, filename: {filename}")

#             X_data = np.concatenate((X_data, data_signal), axis=0)

#             self.X_data_all.append(X_data)
#             self.y_data_all.append(metadata_file['age_months'])
        
#     def on_epoch_end(self):
#         """Updates indexes after each epoch."""

#         # Create new metadata DataFrame with only the current subject IDs
#         if self.metadata_temp is None:
#             self.metadata_temp = self.metadata[self.metadata['code'].isin(self.list_IDs)].reset_index(drop=True)
                               
#         idx_base = np.arange(len(self.metadata_temp))
#         idx_epoch = np.tile(idx_base, self.iter_per_epoch)

#         self.indexes = idx_epoch

#         if self.shuffle == True:
#             np.random.shuffle(self.indexes)

#     def generate_data(self, indexes):
#         """Generates data containing batch_size averaged time series.

#         Args:
#         -------
#         list_IDs_temp: list
#             list of label ids to load

#         return: batch of averaged time series
#         """
#         X_data = np.zeros((0, self.n_channels, self.n_timepoints))
#         y_data = []

#         for index in indexes:
#             X = self.create_averaged_epoch(self.X_data_all[index])

#             X_data = np.concatenate((X_data, X), axis=0)
#             y_data.append(self.y_data_all[index])
            
#         return np.swapaxes(X_data,1,2), np.array(y_data).reshape((-1,1))
    
#     def create_averaged_epoch(self,
#                               data_signal):
#         """
#         Function to create averages of self.n_average epochs.
#         Will create one averaged epoch per found unique label from self.n_average random epochs.

#         Args:
#         --------
#         data_signal: numpy array
#             Data from one person as numpy array
#         """
                                               
#         # Create new data collection:
#         X_data = np.zeros((0, self.n_channels, self.n_timepoints))
#         num_epochs = len(data_signal)
                                               
#         if num_epochs >= self.n_average:
#             select = np.random.choice(num_epochs, self.n_average, replace=False)
#             signal_averaged = np.mean(data_signal[select,:,:], axis=0)

#         else:
#             if self.warnings:
#                 print("Found only", num_epochs, " epochs and will take those!")          
#             signal_averaged = np.mean(data_signal[:,:,:], axis=0)
                                                                                              
#         X_data = np.concatenate([X_data, np.expand_dims(signal_averaged, axis=0)], axis=0)
                                    
#         if self.gaussian_noise != 0.0:
#             X_data += np.random.normal(0, self.gaussian_noise, X_data.shape)

#         return X_data


#     def load_signal(self,
#                     filename):
#         """Load EEG signal from one person.

#         Args:
#         -------
#         filename: str
#             filename...

#         return: loaded array
#         """
#         return zarr.open(os.path.join(filename), mode='r')
%%time 
# IDs_train = [152, 18, 11, 435, 617]
# IDs_val = [180, 105, 19, 332]

train_generator_noise = DataGeneratorReduced(list_IDs = IDs_train,
                                             BASE_PATH = PATH_DATA_PROCESSED_DL_REDUCED,
                                             metadata = df_metadata,
                                             n_average = 3,
                                             batch_size = 10,
                                             gaussian_noise=0.01,
                                             iter_per_epoch = 3,
                                             n_timepoints = 501, 
                                             n_channels=30, 
                                             shuffle=True)

val_generator = DataGeneratorReduced(list_IDs = IDs_val,
                                     BASE_PATH = PATH_DATA_PROCESSED_DL_REDUCED,
                                     metadata = df_metadata,
                                     n_average = 3,
                                     batch_size = 10,
                                     n_timepoints = 501,
                                     iter_per_epoch = 3,
                                     n_channels=30,
                                     shuffle=True)<jupyter_output>CPU times: user 22.5 s, sys: 22 s, total: 44.5 s
Wall time: 1min 17s<jupyter_text>Helper functions<jupyter_code>import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, Input, Sequential
from tensorflow.keras.layers import Bidirectional, LSTM, Dropout, BatchNormalization, Dense, Conv1D, LeakyReLU, AveragePooling1D, Flatten, Reshape, MaxPooling1D
from tensorflow.keras.optimizers import Adam, Adadelta, SGD
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
from tensorflow.keras.metrics import RootMeanSquaredError, MeanAbsoluteError
import time

n_timesteps = 501
n_features = 30 
n_outputs = 1

input_shape = (n_timesteps, n_features)<jupyter_output><empty_output><jupyter_text>Testing of different architectures Below we will test multiple different architectures, most of them as discussed in "Deep learning for time series classification: a review", by <NAME> al (2019). Most of them are inspired again on other papers. Refer to the Ismail Fawaz paper for the original papers.1. Fully-connected NN2. CNN3. ResNet4. Encoder5. Time-CNNOther architectures to test:6. BLSTM-LSTM 7. InceptionTime  1. Fully connected NN<jupyter_code>def fully_connected_model():
    """ Returns the fully connected model from <NAME> et al. (2019). """

    input_layer = keras.layers.Input(input_shape)

    input_layer_flattened = keras.layers.Flatten()(input_layer)

    layer_1 = keras.layers.Dropout(0.1)(input_layer_flattened)
    layer_1 = keras.layers.Dense(500, activation='relu')(layer_1)

    layer_2 = keras.layers.Dropout(0.2)(layer_1)
    layer_2 = keras.layers.Dense(500, activation='relu')(layer_2)

    layer_3 = keras.layers.Dropout(0.2)(layer_2)
    layer_3 = keras.layers.Dense(500, activation='relu')(layer_3)

    output_layer = keras.layers.Dropout(0.3)(layer_3)
    output_layer = keras.layers.Dense(1)(output_layer)

    model = keras.models.Model(inputs=input_layer, outputs=output_layer)

    return model
model = fully_connected_model()

optimizer = Adadelta(learning_rate=0.01)    
              
model.compile(loss='mean_squared_error', 
              optimizer=optimizer, 
              metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

# 01 seems to be incorrect (makes too many predictions, changed model)
# Fully_connected_regressor_01: MSE, Adadelta, N_average=30, 5000 epochs, ES=1000, RLR=200, gaussian=0.01
# Fully_connected_regressor_02: MSE, Adadelta, N_average=30, 5000 epochs, ES=1000, RLR=200, gaussian=0.01
output_filename = 'Fully_connected_regressor_03'
output_file = os.path.join(PATH_MODELS, output_filename)

checkpointer = ModelCheckpoint(filepath = output_file + ".hdf5", monitor='val_loss', verbose=1, save_best_only=True)
earlystopper = EarlyStopping(monitor='val_loss', patience=1000, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=200, min_lr=0.0001, verbose=1)
%%time

epochs = 5000

# fit network
history = model.fit(x=train_generator_noise,
                    validation_data=val_generator,
                    epochs=epochs, 
                    callbacks=[checkpointer, earlystopper, reduce_lr])<jupyter_output>Epoch 1/5000
83/83 [==============================] - 6s 68ms/step - loss: 868.5831 - root_mean_squared_error: 29.4656 - mean_absolute_error: 27.5962 - val_loss: 638.1409 - val_root_mean_squared_error: 25.2614 - val_mean_absolute_error: 23.1626

Epoch 00001: val_loss improved from inf to 638.14087, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/Fully_connected_regressor_03.hdf5
Epoch 2/5000
83/83 [==============================] - 5s 55ms/step - loss: 505.3884 - root_mean_squared_error: 22.4179 - mean_absolute_error: 20.1228 - val_loss: 298.9310 - val_root_mean_squared_error: 17.2896 - val_mean_absolute_error: 14.1719

Epoch 00002: val_loss improved from 638.14087 to 298.93100, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/Fully_connected_regressor_03.hdf5
Epoch 3/5000
83/83 [==============================] - 4s 47ms/step - loss: 272.8383 - root_mean_squared_error: 16.5092 - mean_absolute_error: 13[...]<jupyter_text>2. CNN<jupyter_code>def cnn_model():
    """ Returns the CNN (FCN) model from <NAME> et al. (2019). """

    input_layer = keras.layers.Input(input_shape)

    conv1 = keras.layers.Conv1D(filters=128, kernel_size=8, padding='same')(input_layer)
    conv1 = keras.layers.BatchNormalization()(conv1)
    conv1 = keras.layers.Activation(activation='relu')(conv1)

    conv2 = keras.layers.Conv1D(filters=256, kernel_size=5, padding='same')(conv1)
    conv2 = keras.layers.BatchNormalization()(conv2)
    conv2 = keras.layers.Activation('relu')(conv2)

    conv3 = keras.layers.Conv1D(128, kernel_size=3, padding='same')(conv2)
    conv3 = keras.layers.BatchNormalization()(conv3)
    conv3 = keras.layers.Activation('relu')(conv3)

    gap_layer = keras.layers.GlobalAveragePooling1D()(conv3)

    output_layer = keras.layers.Dense(1)(gap_layer)

    model = keras.models.Model(inputs=input_layer, outputs=output_layer)

    return model 
model = cnn_model()

optimizer = Adam(learning_rate=0.01)    
              
model.compile(loss='mean_squared_error', 
              optimizer=optimizer, 
              metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

# CNN_regressor_01: MSE, Adam, N_average=30, 2000 epochs, ES=250, RLR=50, gaussian=0.01
output_filename = 'CNN_regressor_03'
output_file = os.path.join(PATH_MODELS, output_filename)

checkpointer = ModelCheckpoint(filepath = output_file + ".hdf5", monitor='val_loss', verbose=1, save_best_only=True)
earlystopper = EarlyStopping(monitor='val_loss', patience=250, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=50, min_lr=0.0001, verbose=1)
%%time

epochs = 2000

# fit network
history = model.fit(x=train_generator_noise,
                    validation_data=val_generator,
                    epochs=epochs, 
                    callbacks=[checkpointer, earlystopper, reduce_lr])<jupyter_output>Epoch 1/2000
84/84 [==============================] - ETA: 0s - loss: nan - root_mean_squared_error: nan - mean_absolute_error: nan<jupyter_text>3. ResNet<jupyter_code>def resnet_model():
    """ Returns the ResNet model from <NAME> al. (2019). """
    n_feature_maps = 64

    input_layer = keras.layers.Input(input_shape)

    # BLOCK 1

    conv_x = keras.layers.Conv1D(filters=n_feature_maps, kernel_size=8, padding='same')(input_layer)
    conv_x = keras.layers.BatchNormalization()(conv_x)
    conv_x = keras.layers.Activation('relu')(conv_x)

    conv_y = keras.layers.Conv1D(filters=n_feature_maps, kernel_size=5, padding='same')(conv_x)
    conv_y = keras.layers.BatchNormalization()(conv_y)
    conv_y = keras.layers.Activation('relu')(conv_y)

    conv_z = keras.layers.Conv1D(filters=n_feature_maps, kernel_size=3, padding='same')(conv_y)
    conv_z = keras.layers.BatchNormalization()(conv_z)

    # expand channels for the sum
    shortcut_y = keras.layers.Conv1D(filters=n_feature_maps, kernel_size=1, padding='same')(input_layer)
    shortcut_y = keras.layers.BatchNormalization()(shortcut_y)

    output_block_1 = keras.layers.add([shortcut_y, conv_z])
    output_block_1 = keras.layers.Activation('relu')(output_block_1)

    # BLOCK 2

    conv_x = keras.layers.Conv1D(filters=n_feature_maps * 2, kernel_size=8, padding='same')(output_block_1)
    conv_x = keras.layers.BatchNormalization()(conv_x)
    conv_x = keras.layers.Activation('relu')(conv_x)

    conv_y = keras.layers.Conv1D(filters=n_feature_maps * 2, kernel_size=5, padding='same')(conv_x)
    conv_y = keras.layers.BatchNormalization()(conv_y)
    conv_y = keras.layers.Activation('relu')(conv_y)

    conv_z = keras.layers.Conv1D(filters=n_feature_maps * 2, kernel_size=3, padding='same')(conv_y)
    conv_z = keras.layers.BatchNormalization()(conv_z)

    # expand channels for the sum
    shortcut_y = keras.layers.Conv1D(filters=n_feature_maps * 2, kernel_size=1, padding='same')(output_block_1)
    shortcut_y = keras.layers.BatchNormalization()(shortcut_y)

    output_block_2 = keras.layers.add([shortcut_y, conv_z])
    output_block_2 = keras.layers.Activation('relu')(output_block_2)

    # BLOCK 3

    conv_x = keras.layers.Conv1D(filters=n_feature_maps * 2, kernel_size=8, padding='same')(output_block_2)
    conv_x = keras.layers.BatchNormalization()(conv_x)
    conv_x = keras.layers.Activation('relu')(conv_x)

    conv_y = keras.layers.Conv1D(filters=n_feature_maps * 2, kernel_size=5, padding='same')(conv_x)
    conv_y = keras.layers.BatchNormalization()(conv_y)
    conv_y = keras.layers.Activation('relu')(conv_y)

    conv_z = keras.layers.Conv1D(filters=n_feature_maps * 2, kernel_size=3, padding='same')(conv_y)
    conv_z = keras.layers.BatchNormalization()(conv_z)

    # no need to expand channels because they are equal
    shortcut_y = keras.layers.BatchNormalization()(output_block_2)

    output_block_3 = keras.layers.add([shortcut_y, conv_z])
    output_block_3 = keras.layers.Activation('relu')(output_block_3)

    # FINAL

    gap_layer = keras.layers.GlobalAveragePooling1D()(output_block_3)

    output_layer = keras.layers.Dense(1)(gap_layer)

    model = keras.models.Model(inputs=input_layer, outputs=output_layer)

    return model
model = resnet_model()

optimizer = Adam(learning_rate=0.01)    
              
model.compile(loss='mean_squared_error', 
              optimizer=optimizer, 
              metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

# ResNet_regressor_01: MSE, Adam, N_average=30, 1500 epochs, ES=250, RLR=50, gaussian=0.01
output_filename = 'ResNet_regressor_02'
output_file = os.path.join(PATH_MODELS, output_filename)

checkpointer = ModelCheckpoint(filepath = output_file + ".hdf5", monitor='val_loss', verbose=1, save_best_only=True)
earlystopper = EarlyStopping(monitor='val_loss', patience=250, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=50, min_lr=0.0001, verbose=1)
%%time

epochs = 1500

# fit network
history = model.fit(x=train_generator_noise,
                    validation_data=val_generator,
                    epochs=epochs, 
                    callbacks=[checkpointer, earlystopper, reduce_lr])<jupyter_output>Epoch 1/1500
143/143 [==============================] - 95s 646ms/step - loss: 189.6503 - root_mean_squared_error: 13.4258 - mean_absolute_error: 11.0235 - val_loss: 1908.8988 - val_root_mean_squared_error: 43.6909 - val_mean_absolute_error: 35.3563

Epoch 00001: val_loss improved from inf to 1908.89880, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/ResNet_regressor_01.hdf5
Epoch 2/1500
143/143 [==============================] - 89s 619ms/step - loss: 109.5164 - root_mean_squared_error: 10.4630 - mean_absolute_error: 8.7816 - val_loss: 142.7589 - val_root_mean_squared_error: 11.9482 - val_mean_absolute_error: 9.5680

Epoch 00002: val_loss improved from 1908.89880 to 142.75887, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/ResNet_regressor_01.hdf5
Epoch 3/1500
143/143 [==============================] - 90s 630ms/step - loss: 104.9589 - root_mean_squared_error: 10.2428 - mean_absolute_error: 8.6765 [...]<jupyter_text>4. Encoder<jupyter_code>import tensorflow_addons as tfa

def encoder_model():
    """ Returns the Encoder model from <NAME> et al. (2019). """
    input_layer = keras.layers.Input(input_shape)

    # conv block -1
    conv1 = keras.layers.Conv1D(filters=128,kernel_size=5,strides=1,padding='same')(input_layer)
    conv1 = tfa.layers.InstanceNormalization()(conv1)
    conv1 = keras.layers.PReLU(shared_axes=[1])(conv1)
    conv1 = keras.layers.Dropout(rate=0.2)(conv1)
    conv1 = keras.layers.MaxPooling1D(pool_size=2)(conv1)
    # conv block -2
    conv2 = keras.layers.Conv1D(filters=256,kernel_size=11,strides=1,padding='same')(conv1)
    conv2 = tfa.layers.InstanceNormalization()(conv2)
    conv2 = keras.layers.PReLU(shared_axes=[1])(conv2)
    conv2 = keras.layers.Dropout(rate=0.2)(conv2)
    conv2 = keras.layers.MaxPooling1D(pool_size=2)(conv2)
    # conv block -3
    conv3 = keras.layers.Conv1D(filters=512,kernel_size=21,strides=1,padding='same')(conv2)
    conv3 = tfa.layers.InstanceNormalization()(conv3)
    conv3 = keras.layers.PReLU(shared_axes=[1])(conv3)
    conv3 = keras.layers.Dropout(rate=0.2)(conv3)
    # split for attention
    attention_data = keras.layers.Lambda(lambda x: x[:,:,:256])(conv3)
    attention_softmax = keras.layers.Lambda(lambda x: x[:,:,256:])(conv3)
    # attention mechanism
    attention_softmax = keras.layers.Softmax()(attention_softmax)
    multiply_layer = keras.layers.Multiply()([attention_softmax,attention_data])
    # last layer
    dense_layer = keras.layers.Dense(units=256,activation='sigmoid')(multiply_layer)
    dense_layer = tfa.layers.InstanceNormalization()(dense_layer)
    # output layer
    flatten_layer = keras.layers.Flatten()(dense_layer)
    output_layer = keras.layers.Dense(1)(flatten_layer)

    model = keras.models.Model(inputs=input_layer, outputs=output_layer)

    return model
model = encoder_model()

optimizer = Adam(learning_rate=0.00001)    
              
model.compile(loss='mean_squared_error', 
              optimizer=optimizer, 
              metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

# Encoder_regressor_01: MSE, Adam, N_average=30, 1500 epochs, ES=250, RLR=50, gaussian=0.01 (LR = 0.0001, no reduction)
output_filename = 'Encoder_regressor_02'
output_file = os.path.join(PATH_MODELS, output_filename)

checkpointer = ModelCheckpoint(filepath = output_file + ".hdf5", monitor='val_loss', verbose=1, save_best_only=True)
%%time

epochs = 100

# fit network
history = model.fit(x=train_generator_noise,
                    validation_data=val_generator,
                    epochs=epochs, 
                    callbacks=[checkpointer])<jupyter_output>Epoch 1/100
143/143 [==============================] - 97s 669ms/step - loss: 772.3317 - root_mean_squared_error: 27.7892 - mean_absolute_error: 25.8278 - val_loss: 711.6334 - val_root_mean_squared_error: 26.6765 - val_mean_absolute_error: 24.4745

Epoch 00001: val_loss improved from inf to 711.63336, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/Encoder_regressor_01.hdf5
Epoch 2/100
143/143 [==============================] - 93s 654ms/step - loss: 585.1881 - root_mean_squared_error: 24.1294 - mean_absolute_error: 21.6805 - val_loss: 317.0688 - val_root_mean_squared_error: 17.8064 - val_mean_absolute_error: 15.0762

Epoch 00002: val_loss improved from 711.63336 to 317.06882, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/Encoder_regressor_01.hdf5
Epoch 3/100
143/143 [==============================] - 92s 641ms/step - loss: 192.9793 - root_mean_squared_error: 13.8633 - mean_absolute_error: 11.2211 -[...]<jupyter_text>5. Time-CNN<jupyter_code># https://github.com/hfawaz/dl-4-tsc/blob/master/classifiers/cnn.py

def timecnn_model():
    """ Returns the Time-CNN model from <NAME> et al. (2019). """
    
    padding = 'valid'
    input_layer = keras.layers.Input(input_shape)

    conv1 = keras.layers.Conv1D(filters=6,kernel_size=7,padding=padding,activation='sigmoid')(input_layer)
    conv1 = keras.layers.AveragePooling1D(pool_size=3)(conv1)

    conv2 = keras.layers.Conv1D(filters=12,kernel_size=7,padding=padding,activation='sigmoid')(conv1)
    conv2 = keras.layers.AveragePooling1D(pool_size=3)(conv2)

    flatten_layer = keras.layers.Flatten()(conv2)

    output_layer = keras.layers.Dense(units=1)(flatten_layer)

    model = keras.models.Model(inputs=input_layer, outputs=output_layer)

    return model
model = cnn_model()

optimizer = Adam(learning_rate=0.01)    
              
model.compile(loss='mean_squared_error', 
              optimizer=optimizer, 
              metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

# TimeCNN_regressor_01: MSE, Adam, N_average=30, 2000 epochs, ES=250, RLR=50, gaussian=0.01
output_filename = 'TimeCNN_regressor_02'
output_file = os.path.join(PATH_MODELS, output_filename)

checkpointer = ModelCheckpoint(filepath = output_file + ".hdf5", monitor='val_loss', verbose=1, save_best_only=True)
earlystopper = EarlyStopping(monitor='val_loss', patience=250, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=50, min_lr=0.0001, verbose=1)
%%time

epochs = 2000

# fit network
history = model.fit(x=train_generator_noise,
                    validation_data=val_generator,
                    epochs=epochs, 
                    callbacks=[checkpointer, earlystopper, reduce_lr])<jupyter_output>Epoch 1/2000
143/143 [==============================] - 93s 642ms/step - loss: 321.7530 - root_mean_squared_error: 17.4269 - mean_absolute_error: 14.5173 - val_loss: 160.2513 - val_root_mean_squared_error: 12.6590 - val_mean_absolute_error: 10.2964

Epoch 00001: val_loss improved from inf to 160.25128, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/TimeCNN_regressor_01.hdf5
Epoch 2/2000
143/143 [==============================] - 89s 623ms/step - loss: 99.1723 - root_mean_squared_error: 9.9552 - mean_absolute_error: 8.2079 - val_loss: 187.8685 - val_root_mean_squared_error: 13.7065 - val_mean_absolute_error: 10.9276

Epoch 00002: val_loss did not improve from 160.25128
Epoch 3/2000
143/143 [==============================] - 89s 622ms/step - loss: 105.3432 - root_mean_squared_error: 10.2628 - mean_absolute_error: 8.5573 - val_loss: 121.3760 - val_root_mean_squared_error: 11.0171 - val_mean_absolute_error: 9.1967

Epoch 00003: val_loss improved [...]<jupyter_text>6. BLSTM-LSTM<jupyter_code>def blstm_lstm_model():
    """ Returns the BLSTM-LSTM model from Kaushik et al. (2019). """

    # MARK: This model compresses too much in the last phase, check if possible to improve.

    model = keras.Sequential()

    # BLSTM layer
    model.add(Bidirectional(LSTM(256, return_sequences=True), input_shape=input_shape))
    model.add(Dropout(.2))
    model.add(BatchNormalization())

    # LSTM layer
    model.add(LSTM(128, return_sequences=True))
    model.add(BatchNormalization())

    # LSTM layer
    model.add(LSTM(64, return_sequences=False))
    model.add(BatchNormalization())

    # Fully connected layer
    model.add(Dense(32))

    model.add(Dense(n_outputs))

    return model 
model = blstm_lstm_model()

optimizer = Adam(learning_rate=0.01)    
              
model.compile(loss='mean_squared_error', 
              optimizer=optimizer, 
              metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

# BLSTM_regressor_01: MSE, Adam, N_average=30, 1500 epochs, ES=250, RLR=50, gaussian=0.01
output_filename = 'BLSTM_regressor_01'
output_file = os.path.join(PATH_MODELS, output_filename)

checkpointer = ModelCheckpoint(filepath = output_file + ".hdf5", monitor='val_loss', verbose=1, save_best_only=True)
earlystopper = EarlyStopping(monitor='val_loss', patience=250, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=50, min_lr=0.0001, verbose=1)

model.summary()
%%time

epochs = 1500

# fit network
history = model.fit(x=train_generator_noise,
                    validation_data=val_generator,
                    epochs=epochs, 
                    callbacks=[checkpointer, earlystopper, reduce_lr])<jupyter_output>Epoch 1/1500
83/83 [==============================] - 81s 973ms/step - loss: 226.7901 - root_mean_squared_error: 15.0596 - mean_absolute_error: 12.0603 - val_loss: 277.5875 - val_root_mean_squared_error: 16.6610 - val_mean_absolute_error: 14.4406

Epoch 00001: val_loss improved from inf to 277.58746, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/BLSTM_regressor_01.hdf5
Epoch 2/1500
61/83 [=====================>........] - ETA: 24s - loss: 131.2852 - root_mean_squared_error: 11.4580 - mean_absolute_error: 9.6893<jupyter_text>7. InceptionTime<jupyter_code>class Regressor_Inception:

    def __init__(self, output_directory, input_shape, verbose=False, build=True, batch_size=64,
                 nb_filters=32, use_residual=True, use_bottleneck=True, depth=6, kernel_size=41, nb_epochs=1500):

        self.output_directory = output_directory

        self.nb_filters = nb_filters
        self.use_residual = use_residual
        self.use_bottleneck = use_bottleneck
        self.depth = depth
        self.kernel_size = kernel_size - 1
        self.callbacks = None
        self.batch_size = batch_size
        self.bottleneck_size = 32
        self.nb_epochs = nb_epochs

        if build == True:
            self.model = self.build_model(input_shape)
            if (verbose == True):
                self.model.summary()
            self.verbose = verbose
            self.model.save_weights(self.output_directory + 'inception_model_init.hdf5')

    def _inception_module(self, input_tensor, stride=1, activation='linear'):

        if self.use_bottleneck and int(input_tensor.shape[-1]) > 1:
            input_inception = tf.keras.layers.Conv1D(filters=self.bottleneck_size, kernel_size=1,
                                                  padding='same', activation=activation, use_bias=False)(input_tensor)
        else:
            input_inception = input_tensor

        # kernel_size_s = [3, 5, 8, 11, 17]
        kernel_size_s = [self.kernel_size // (2 ** i) for i in range(3)]

        conv_list = []

        for i in range(len(kernel_size_s)):
            conv_list.append(tf.keras.layers.Conv1D(filters=self.nb_filters, kernel_size=kernel_size_s[i],
                                                 strides=stride, padding='same', activation=activation, use_bias=False)(
                input_inception))

        max_pool_1 = tf.keras.layers.MaxPool1D(pool_size=3, strides=stride, padding='same')(input_tensor)

        conv_6 = tf.keras.layers.Conv1D(filters=self.nb_filters, kernel_size=1,
                                     padding='same', activation=activation, use_bias=False)(max_pool_1)

        conv_list.append(conv_6)

        x = tf.keras.layers.Concatenate(axis=2)(conv_list)
        x = tf.keras.layers.BatchNormalization()(x)
        x = tf.keras.layers.Activation(activation='relu')(x)
        return x

    def _shortcut_layer(self, input_tensor, out_tensor):
        shortcut_y = tf.keras.layers.Conv1D(filters=int(out_tensor.shape[-1]), kernel_size=1,
                                         padding='same', use_bias=False)(input_tensor)
        shortcut_y = tf.keras.layers.BatchNormalization()(shortcut_y)

        x = tf.keras.layers.Add()([shortcut_y, out_tensor])
        x = tf.keras.layers.Activation('relu')(x)
        return x

    def build_model(self, input_shape):
        input_layer = tf.keras.layers.Input(input_shape)

        x = input_layer
        input_res = input_layer

        for d in range(self.depth):

            x = self._inception_module(x)

            if self.use_residual and d % 3 == 2:
                x = self._shortcut_layer(input_res, x)
                input_res = x

        pooling_layer = tf.keras.layers.AveragePooling1D(pool_size=50)(x)
        flat_layer = tf.keras.layers.Flatten()(pooling_layer)
        dense_layer = tf.keras.layers.Dense(128, activation='relu')(flat_layer)
        output_layer = tf.keras.layers.Dense(1)(dense_layer)

        model = tf.keras.models.Model(inputs=input_layer, outputs=output_layer)

        return model
model = Regressor_Inception(PATH_MODELS, input_shape, verbose=True).model

optimizer = Adam(learning_rate=0.01)   
              
model.compile(loss='mean_squared_error', 
              optimizer=optimizer, 
              metrics=[RootMeanSquaredError(), MeanAbsoluteError()])

# 'Inception_regressor_01' (n_average = 40, gaussian_noise = 0.01, MAE)
# 'Inception_regressor_02' (n_average = 1, gaussian_noise = 0.01, MAE)
# 'Inception_regressor_03' (n_average = 40, gaussian_noise = 0.01, MSE)
# 'Inception_regressor_04' (n_average = 1, gaussian_noise = 0.01, MSE)
# 'Inception_regressor_05' (n_average = 100, gaussian_noise = 0.01, MAE)
output_filename = 'Inception_regressor_05'
output_file = os.path.join(PATH_MODELS, output_filename)

checkpointer = ModelCheckpoint(filepath = output_file + ".hdf5", monitor='val_loss', verbose=1, save_best_only=True)
earlystopper = EarlyStopping(monitor='val_loss', patience=100, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=20, min_lr=0.0001, verbose=1)
%%time

epochs = 1500

# fit network
history = model.fit(x=train_generator_noise,
                    validation_data=val_generator,
                    epochs=epochs,
                    callbacks = [checkpointer, earlystopper, reduce_lr])<jupyter_output>Epoch 1/1500
143/143 [==============================] - 249s 2s/step - loss: 14.8795 - root_mean_squared_error: 21.7512 - mean_absolute_error: 14.8795 - val_loss: 14.4589 - val_root_mean_squared_error: 18.1789 - val_mean_absolute_error: 14.4589

Epoch 00001: val_loss improved from inf to 14.45891, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/Inception_regressor_05.hdf5
Epoch 2/1500
143/143 [==============================] - 238s 2s/step - loss: 8.9878 - root_mean_squared_error: 11.2585 - mean_absolute_error: 8.9878 - val_loss: 8.4816 - val_root_mean_squared_error: 10.7197 - val_mean_absolute_error: 8.4816

Epoch 00002: val_loss improved from 14.45891 to 8.48164, saving model to /Users/bbruns/Desktop/Studie/Thesis/Code/EEG_age_prediction/trained_models/Inception_regressor_05.hdf5
Epoch 3/1500
143/143 [==============================] - 241s 2s/step - loss: 7.9297 - root_mean_squared_error: 9.8073 - mean_absolute_error: 7.9297 - val_loss: 8.62[...]<jupyter_text>Compare models  Helper functions for comparison<jupyter_code># 'Inception_regressor_01' (n_average = 40, gaussian_noise = 0.01, MAE)
model_path = os.path.join(PATH_MODELS, 'Inception_regressor_01.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# 'Inception_regressor_02' (n_average = 1, gaussian_noise = 0.01, MAE)
model_path = os.path.join(PATH_MODELS, 'Inception_regressor_02.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# 'Inception_regressor_03' (n_average = 40, gaussian_noise = 0.01, MSE)
model_path = os.path.join(PATH_MODELS, 'Inception_regressor_03.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# 'Inception_regressor_04' (n_average = 1, gaussian_noise = 0.01, MSE)
model_path = os.path.join(PATH_MODELS, 'Inception_regressor_04.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# 'Inception_regressor_05' (n_average = 100, gaussian_noise = 0.01, MAE)
model_path = os.path.join(PATH_MODELS, 'Inception_regressor_05.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# Fully_connected_regressor_02: MSE, Adadelta, N_average=30, 5000 epochs, ES=1000, RLR=200, gaussian=0.01
model_path = os.path.join(PATH_MODELS, 'Fully_connected_regressor_02.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# CNN_regressor_01: MSE, Adam, N_average=30, 2000 epochs, ES=250, RLR=50, gaussian=0.01
model_path = os.path.join(PATH_MODELS, 'CNN_regressor_01.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# ResNet_regressor_01: MSE, Adam, N_average=30, 1500 epochs, ES=250, RLR=50, gaussian=0.01
model_path = os.path.join(PATH_MODELS, 'ResNet_regressor_01.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# Encoder_regressor_01: MSE, Adam, N_average=30, 1500 epochs, ES=250, RLR=50, gaussian=0.01 (LR = 0.0001, no reduction)
model_path = os.path.join(PATH_MODELS, 'Encoder_regressor_01.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)
# TimeCNN_regressor_01: MSE, Adam, N_average=30, 2000 epochs, ES=250, RLR=50, gaussian=0.01
model_path = os.path.join(PATH_MODELS, 'TimeCNN_regressor_01.hdf5')

loaded_model = tf.keras.models.load_model(model_path)
evaluate_model(loaded_model)<jupyter_output>143/143 [==============================] - 65s 452ms/step - loss: 43.6008 - root_mean_squared_error: 6.6031 - mean_absolute_error: 5.1059
30/30 [==============================] - 14s 453ms/step - loss: 56.7583 - root_mean_squared_error: 7.5338 - mean_absolute_error: 6.0041
35/35 [==============================] - 16s 473ms/step - loss: 62.2474 - root_mean_squared_error: 7.8897 - mean_absolute_error: 6.1511
{'fraction_non_alphanumeric': 0.1218059847742519, 'fraction_numerical': 0.04999874375015703, 'mean_word_length': 3.416555703506436, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 42, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '13433', 'license': 'Apache-2.0', 'n_tokens_mistral': 14799, 'n_tokens_neox': 12988, 'n_words': 2579, 'path': 'Notebooks/Deep learning EEG_model testing_DL_REDUCED.ipynb', 'repo_name': 'svenvanderburg/EEG_age_prediction', 'size': '39843'}