<jupyter_start><jupyter_text>Install Kaggle API<jupyter_code>!pip install kaggle
!pip install --upgrade --force-reinstall --no-deps kaggle # force install the latest version<jupyter_output>Requirement already satisfied: kaggle in /usr/local/lib/python3.7/dist-packages (1.5.10)
Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from kaggle) (1.24.3)
Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from kaggle) (2020.12.5)
Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from kaggle) (4.41.1)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from kaggle) (2.23.0)
Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.7/dist-packages (from kaggle) (1.15.0)
Requirement already satisfied: python-slugify in /usr/local/lib/python3.7/dist-packages (from kaggle) (4.0.1)
Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from kaggle) (2.8.1)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->kaggle) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 [...]<jupyter_text>API CredentialsTo use the Kaggle API, sign up for a Kaggle account at https://www.kaggle.com. Then go to the 'Account' tab of your user profile (https://www.kaggle.com/username/account) and select 'Create API Token'. This will trigger the download of kaggle.json, a file containing your API credentials. Place this file on your Google Drive anywhere.With the next snippet you download your credentials to Colab and you can start using Kaggle API<jupyter_code># Install gcloud python module and google-api-python-client module
!pip install --upgrade gcloud
!pip install --upgrade google-api-python-client
from googleapiclient.discovery import build
import io, os
from googleapiclient.http import MediaIoBaseDownload
from google.colab import auth

auth.authenticate_user()

drive_service = build('drive', 'v3')
results = drive_service.files().list(
        q="name = 'kaggle.json'", fields="files(id)").execute()
kaggle_api_key = results.get('files', [])

filename = "/content/.kaggle/kaggle.json"
os.makedirs(os.path.dirname(filename), exist_ok=True)

request = drive_service.files().get_media(fileId=kaggle_api_key[0]['id'])
fh = io.FileIO(filename, 'wb')
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
    status, done = downloader.next_chunk()
    print("Download %d%%." % int(status.progress() * 100))
os.chmod(filename, 600)
!mkdir ~/.kaggle
!cp /content/.kaggle/kaggle.json ~/.kaggle/kaggle.json<jupyter_output><empty_output><jupyter_text>Download The Dataset From Kaggle<jupyter_code>!kaggle competitions download -c ranzcr-clip-catheter-line-classification -p /content/
!unzip -q /content/ranzcr-clip-catheter-line-classification.zip
!rm /content/ranzcr-clip-catheter-line-classification.zip
%cd /content
!ls<jupyter_output>/content
adc.json	       test	       train_annotations.csv
sample_data	       test_tfrecords  train.csv
sample_submission.csv  train	       train_tfrecords<jupyter_text>Download the latest EfficientNet weights<jupyter_code>!wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/noisystudent/noisy_student_efficientnet-b3.tar.gz
!tar -xf noisy_student_efficientnet-b3.tar.gz
!wget https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/python/keras/applications/efficientnet_weight_update_util.py
!python efficientnet_weight_update_util.py --model b3 --notop --ckpt noisy-student-efficientnet-b3/model.ckpt --o efficientnetb3_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb0_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb1_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb2_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb3_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb4_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb5_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb6_notop.h5
!wget https://storage.googleapis.com/keras-applications/efficientnetb7_notop.h5<jupyter_output>--2021-03-11 08:58:05--  https://storage.googleapis.com/keras-applications/efficientnetb0_notop.h5
Resolving storage.googleapis.com (storage.googleapis.com)... 172.16.58.3, 172.16.17.32, 172.16.17.32, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.16.58.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16705208 (16M) [application/x-hdf]
Saving to: ‘efficientnetb0_notop.h5’

efficientnetb0_noto 100%[===================>]  15.93M  68.8MB/s    in 0.2s    

2021-03-11 08:58:06 (68.8 MB/s) - ‘efficientnetb0_notop.h5’ saved [16705208/16705208]

--2021-03-11 08:58:06--  https://storage.googleapis.com/keras-applications/efficientnetb1_notop.h5
Resolving storage.googleapis.com (storage.googleapis.com)... 172.16.31.10, 172.16.17.32, 192.168.127.12, ...
Connecting to storage.googleapis.com (storage.googleapis.com)|172.16.31.10|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27018416 (26M) [application/[...]<jupyter_text>Enabling and testing the TPU

First, you'll need to enable TPUs for the notebook:

- Navigate to Edit→Notebook Settings
- select TPU from the Hardware Accelerator drop-down

Next, we'll check that we can connect to the TPU:<jupyter_code>%tensorflow_version 2.x
import tensorflow as tf
print("Tensorflow version " + tf.__version__)
import os

def auto_select_accelerator(tpu_worker_address=None):
    try:
        tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu_worker_address) # TPU detection
        tf.config.experimental_connect_to_cluster(tpu)
        tf.tpu.experimental.initialize_tpu_system(tpu)
        strategy = tf.distribute.TPUStrategy(tpu)
        print("Running on TPU:", tpu.master())
    except ValueError:
        print('Not connected to a TPU runtime; please see the previous cell in this notebook for instructions!')
        strategy = tf.distribute.get_strategy()
    print(f"Running on {strategy.num_replicas_in_sync} replicas")
    
    return strategy

strategy = auto_select_accelerator()
# Listing Devices
from tensorflow.python.client import device_lib
device_lib.list_local_devices()
# Testing for GPU
tf.test.gpu_device_name()<jupyter_output><empty_output><jupyter_text>Import libraries<jupyter_code># http://scikit.ml/stratification.html
# http://scikit.ml/api/skmultilearn.model_selection.iterative_stratification.html#skmultilearn.model_selection.iterative_stratification.iterative_train_test_split
!pip install scikit-multilearn
!pip install arff
# https://www.tensorflow.org/addons
!pip install tensorflow-addons
import numpy as np
import pandas as pd

import gc
import os
from glob import glob
from random import shuffle
import cv2
import datetime
import matplotlib.pyplot as plt

from albumentations import Compose, RandomRotate90, Transpose, Flip, OneOf, CLAHE, IAASharpen, IAAEmboss, RandomBrightnessContrast, JpegCompression, Blur, GaussNoise, HueSaturationValue, ShiftScaleRotate, Normalize

from sklearn.model_selection import train_test_split, KFold
from skmultilearn.model_selection import iterative_train_test_split, IterativeStratification

import tensorflow as tf
import tensorflow_addons as tfa
from tensorflow.keras import Model
from tensorflow.keras.applications import InceptionV3, NASNetLarge, InceptionResNetV2, Xception, EfficientNetB0, EfficientNetB1, EfficientNetB2, EfficientNetB3, EfficientNetB4, EfficientNetB5, EfficientNetB6, EfficientNetB7, ResNet152V2
from tensorflow.keras.losses import binary_crossentropy
from tensorflow.keras.callbacks import Callback, ModelCheckpoint, EarlyStopping, ReduceLROnPlateau, TerminateOnNaN, LearningRateScheduler
from tensorflow.keras.layers import Lambda, Reshape, DepthwiseConv2D, ZeroPadding2D, Add, MaxPooling2D,Activation, Flatten, Conv2D, Dense, Input, Dropout, Concatenate, GlobalMaxPooling2D, GlobalAveragePooling2D, BatchNormalization
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import regularizers
from tensorflow.keras import backend as K

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'<jupyter_output><empty_output><jupyter_text>For reproducibility:<jupyter_code>SEED = 42

os.environ['PYTHONHASHSEED'] = str(SEED)
np.random.seed(SEED)
tf.random.set_seed(SEED)<jupyter_output><empty_output><jupyter_text>Functions  Build Dataset<jupyter_code>def build_decoder(target_size, with_labels=True, normalize=True, ext='jpg'):
    def decode(path):
        file_bytes = tf.io.read_file(path) # Reads and outputs the entire contents of the input filename.

        if ext == 'png':
            img = tf.image.decode_png(file_bytes, channels=3) # Decode a PNG-encoded image to a uint8 or uint16 tensor
        elif ext in ['jpg', 'jpeg']:
            img = tf.image.decode_jpeg(file_bytes, channels=3) # Decode a JPEG-encoded image to a uint8 tensor
        else:
            raise ValueError("Image extension not supported")

        if normalize:
          img = tf.cast(img, tf.float32) / 255.0 # Casts a tensor to the type float32 and divides by 255.
        else:
          img = tf.cast(img, tf.float32)
        img = tf.image.resize(img, target_size) # Resizing to target size
        return img
    
    def decode_with_labels(path, label):
        return decode(path), tf.dtypes.cast(label, tf.float32)
    
    return decode_with_labels if with_labels else decode

def build_augmenter(with_labels=True):
  """
  When you don't have a large image dataset, it's a good practice 
  to artificially introduce sample diversity by applying random, yet realistic,
  transformations to the training images, such as rotation and 
  horizontal flipping. This helps expose the model to different aspects 
  of the training data and reduce overfitting.
  """
  def augment(img):
      img = tf.image.random_flip_left_right(img)
      img = tf.image.random_flip_up_down(img)
      img = tf.image.random_saturation(img, 0.8, 1.2)
      img = tf.image.random_brightness(img, 0.2)
      img = tf.image.random_contrast(img, 0.8, 1.2)
      img = tf.image.random_hue(img, 0.2)
      return img
  
  def augment_with_labels(img, label):
      return augment(img), label
  
  return augment_with_labels if with_labels else augment

def build_dataset(paths, labels=None, bsize=32, cache=True,
                  decode_fn=None, augment_fn=None,
                  augment=True, repeat=True, shuffle=1024, 
                  cache_dir=""):
    if cache_dir != "" and cache is True:
        os.makedirs(cache_dir, exist_ok=True)
    
    if decode_fn is None:
        decode_fn = build_decoder(labels is not None)
    
    if augment_fn is None:
        augment_fn = build_augmenter(labels is not None)
    
    AUTO = tf.data.experimental.AUTOTUNE
    slices = paths if labels is None else (paths, labels)
    # Better performance with the tf.data API
    # https://www.tensorflow.org/guide/data_performance
    dset = tf.data.Dataset.from_tensor_slices(slices)
    dset = dset.map(decode_fn, num_parallel_calls=AUTO)
    dset = dset.cache(cache_dir) if cache else dset
    dset = dset.map(augment_fn, num_parallel_calls=AUTO) if augment else dset
    dset = dset.repeat() if repeat else dset
    dset = dset.shuffle(shuffle) if shuffle else dset
    # Use buffered prefetching to load images from disk without having I/O become blocking.
    dset = dset.batch(bsize).prefetch(AUTO)
    return dset

def build_tta(paths, bsize=32, decode_fn = None, augment_fn = None, tta = False):
  
  AUTO = tf.data.experimental.AUTOTUNE
  dset = tf.data.Dataset.from_tensor_slices(paths)
  dset = dset.map(decode_fn, num_parallel_calls = AUTO)
  dset = dset.map(augment_fn, num_parallel_calls = AUTO)
  if tta:
    dset = dset.repeat() 
  dset = dset.batch(bsize)
  # Use buffered prefetching to load images from disk without having I/O become blocking.
  dset = dset.prefetch(AUTO)
  return dset<jupyter_output><empty_output><jupyter_text>Model<jupyter_code>def get_model(architecture, img_size, n_classes):
  # At the time of writing, Keras ships with ten of these pre-trained models 
  # already built into the library:
  # https://keras.io/api/applications/

  if architecture == "Xception":
    net = Xception(include_top=False,input_shape=(img_size, img_size, 3), weights='imagenet')
  elif architecture == "InceptionV3":
    net = InceptionV3(include_top=False,input_shape=(img_size, img_size, 3), weights='imagenet')
  elif architecture == "ResNet152V2":
    net = ResNet152V2(include_top=False,input_shape=(img_size, img_size, 3), weights='imagenet')
  elif architecture == "NASNetLarge":
    net = NASNetLarge(include_top=False,input_shape=(img_size, img_size, 3), weights='imagenet')
  elif architecture == "InceptionResNetV2":
    net = InceptionResNetV2(include_top=False,input_shape=(img_size, img_size, 3), weights='imagenet')
  elif architecture == "EfficientNetB0":
    net = EfficientNetB0(include_top=False, weights=None, input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb0_notop.h5')
    net.trainable = True
  elif architecture == "EfficientNetB1":
    net = EfficientNetB1(include_top=False, weights=None,input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb1_notop.h5')
    net.trainable = True
  elif architecture == "EfficientNetB2":
    net = EfficientNetB2(include_top=False, weights=None,input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb2_notop.h5')
    net.trainable = True
  elif architecture == "EfficientNetB3":
    net = EfficientNetB3(include_top=False, weights=None,input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb3_notop.h5')
    net.trainable = True
  elif architecture == "EfficientNetB4":
    net = EfficientNetB4(include_top=False, weights=None,input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb4_notop.h5')
    net.trainable = True
  elif architecture == "EfficientNetB5":
    net = EfficientNetB5(include_top=False, weights=None,input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb5_notop.h5')
    net.trainable = True
  elif architecture == "EfficientNetB6":
    net = EfficientNetB6(include_top=False, weights=None,input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb6_notop.h5')
    net.trainable = True
  elif architecture == "EfficientNetB7":
    net = EfficientNetB7(include_top=False, weights=None,input_shape=(img_size, img_size, 3), drop_connect_rate=0.4)
    net.load_weights('/content/efficientnetb7_notop.h5')
    net.trainable = True
  else:
     raise BaseException(f"The architecture {architecture} is not supported.")

  x = net.output

  # Rebuild top
  x = GlobalAveragePooling2D()(x)
  x = Dropout(0.5)(x)
  output = Dense(n_classes, activation='sigmoid')(x)
  model = Model(inputs=net.inputs, outputs=output)

  return model

def compile_model(strategy, architecture, img_size, n_classes):
  with strategy.scope():
    model = get_model(architecture, img_size, n_classes)
    model.compile(tf.optimizers.Adam(lr=1e-4),
                  loss=tfa.losses.SigmoidFocalCrossEntropy(alpha = 0.50,
                                                           gamma = 2.0),
                  metrics=[tf.keras.metrics.AUC(name="auc", multi_label=True)])
    return model

def get_callbacks(model_path, lr_strategy=None):

  callbacks = []

  callbacks.append(TerminateOnNaN())

  callbacks.append(EarlyStopping(
        monitor='val_loss',
        patience=5,
        mode='min',
        verbose=1,
        restore_best_weights=True))

  callbacks.append(ModelCheckpoint(
        model_path,
        monitor='val_loss',
        verbose=1,
        save_best_only=True,
        mode='min'))

  callbacks.append(ReduceLROnPlateau(
        monitor='val_loss', 
        factor=0.8, 
        patience=2, 
        verbose=1, 
        mode='auto', 
        min_delta=0.0001, 
        cooldown=5, 
        min_lr=0.00001))
  
  if lr_strategy is not None:
    callbacks.append(lr_strategy)

  return callbacks<jupyter_output><empty_output><jupyter_text>LR Finder<jupyter_code>class LRFinder(Callback):
    """
    A simple callback for finding the optimal learning rate range for your model + dataset.

    # Usage
        ```python
            lr_finder = LRFinder(min_lr=1e-5,
                                 max_lr=1e-2,
                                 steps_per_epoch=np.ceil(epoch_size/batch_size),
                                 epochs=3)
            model.fit(X_train, Y_train, callbacks=[lr_finder])

            lr_finder.plot_loss()
        ```

    # Arguments
        min_lr: The lower bound of the learning rate range for the experiment.
        max_lr: The upper bound of the learning rate range for the experiment.
        steps_per_epoch: Number of mini-batches in the dataset. Calculated as `np.ceil(epoch_size/batch_size)`.
        epochs: Number of epochs to run experiment. Usually between 2 and 4 epochs is sufficient.

    # References
        Blog post: jeremyjordan.me/nn-learning-rate
        Original paper: https://arxiv.org/abs/1506.01186
    """

    def __init__(self, min_lr=1e-5, max_lr=1e-2, steps_per_epoch=None, epochs=None):
        super().__init__()

        self.min_lr = min_lr
        self.max_lr = max_lr
        self.total_iterations = steps_per_epoch * epochs
        self.iteration = 0
        self.history = {}

    def clr(self):
        """

        Calculate the learning rate.

        :return:

        """
        x = self.iteration / self.total_iterations
        return self.min_lr + (self.max_lr - self.min_lr) * x

    def on_train_begin(self, logs=None):
        """

        :param logs:
        :return:
        """
        '''Initialize the learning rate to the minimum value at the start of training.'''
        logs = logs or {}
        K.set_value(self.model.optimizer.lr, self.min_lr)

    def on_batch_end(self, epoch, logs=None):
        """
        Record previous batch statistics and update the learning rate.
        :param epoch:
        :param logs:
        :return:
        """
        logs = logs or {}
        self.iteration += 1

        self.history.setdefault('lr', []).append(K.get_value(self.model.optimizer.lr))
        self.history.setdefault('iterations', []).append(self.iteration)

        for k, v in logs.items():
            self.history.setdefault(k, []).append(v)

        K.set_value(self.model.optimizer.lr, self.clr())

    def plot_lr(self, results_dir=""):
        """
        Helper function to quickly inspect the learning rate schedule.
        :return:
        """
        fig_acc = plt.figure(figsize=(10, 10))
        plt.plot(self.history['iterations'], self.history['lr'])
        plt.yscale('log')
        plt.xlabel('Iteration')
        plt.ylabel('Learning rate')
        plt.show()
        fig_acc.savefig(results_dir + "/model_lr.png")

        plt.cla()
        plt.close()

    def plot_loss(self, results_dir=""):
        """
        Helper function to quickly observe the learning rate experiment results
        :return:
        """
        fig_acc = plt.figure(figsize=(10, 10))
        plt.plot(self.history['lr'], self.history['loss'])
        plt.xscale('log')
        plt.xlabel('Learning rate')
        plt.ylabel('Loss')
        plt.show()
        fig_acc.savefig(results_dir + "/model_lr_loss.png")

        plt.cla()
        plt.close()<jupyter_output><empty_output><jupyter_text>Cosine annealing learning rate scheduler with periodic restarts<jupyter_code>class SGDRScheduler(Callback):
    """
    Cosine annealing learning rate scheduler with periodic restarts.
    # Usage
        ```python
            schedule = SGDRScheduler(min_lr=1e-5,
                                     max_lr=1e-2,
                                     steps_per_epoch=np.ceil(epoch_size/batch_size),
                                     lr_decay=0.9,
                                     cycle_length=5,
                                     mult_factor=1.5)
            model.fit(X_train, Y_train, epochs=100, callbacks=[schedule])
        ```
    # Arguments
        min_lr: The lower bound of the learning rate range for the experiment.
        max_lr: The upper bound of the learning rate range for the experiment.
        steps_per_epoch: Number of mini-batches in the dataset. Calculated as `np.ceil(epoch_size/batch_size)`.
        lr_decay: Reduce the max_lr after the completion of each cycle.
                  Ex. To reduce the max_lr by 20% after each cycle, set this value to 0.8.
        cycle_length: Initial number of epochs in a cycle.
        mult_factor: Scale epochs_to_restart after each full cycle completion.
    # References
        Blog post: jeremyjordan.me/nn-learning-rate
        Original paper: http://arxiv.org/abs/1608.03983
    """
    def __init__(self,
                 min_lr,
                 max_lr,
                 steps_per_epoch,
                 lr_decay=1,
                 cycle_length=10,
                 mult_factor=2):

        self.min_lr = min_lr
        self.max_lr = max_lr
        self.lr_decay = lr_decay

        self.batch_since_restart = 0
        self.next_restart = cycle_length

        self.steps_per_epoch = steps_per_epoch

        self.cycle_length = cycle_length
        self.mult_factor = mult_factor

        self.best_weights = None

        self.history = {}

    def clr(self):
        """Calculate the learning rate."""
        fraction_to_restart = self.batch_since_restart / (self.steps_per_epoch * self.cycle_length)
        lr = self.min_lr + 0.5 * (self.max_lr - self.min_lr) * (1 + np.cos(fraction_to_restart * np.pi))
        return lr

    def on_train_begin(self, logs={}):
        """Initialize the learning rate to the minimum value at the start of training."""
        logs = logs or {}
        K.set_value(self.model.optimizer.lr, self.max_lr)

    def on_batch_end(self, batch, logs={}):
        """Record previous batch statistics and update the learning rate."""
        logs = logs or {}
        self.history.setdefault('lr', []).append(K.get_value(self.model.optimizer.lr))
        for k, v in logs.items():
            self.history.setdefault(k, []).append(v)

        self.batch_since_restart += 1
        K.set_value(self.model.optimizer.lr, self.clr())

    def on_epoch_end(self, epoch, logs={}):
        """Check for end of current cycle, apply restarts when necessary."""
        if epoch + 1 == self.next_restart:
            self.batch_since_restart = 0
            self.cycle_length = np.ceil(self.cycle_length * self.mult_factor)
            self.next_restart += self.cycle_length
            self.max_lr *= self.lr_decay
            self.best_weights = self.model.get_weights()

    def on_train_end(self, logs={}):
        """Set weights to the values from the end of the most recent cycle for best performance."""
        self.model.set_weights(self.best_weights)

    def plot_lr(self, results_dir):
        """
        Helper function to quickly observe the learning rate.
        :return:
        """
        fig_acc = plt.figure(figsize=(10, 10))
        plt.plot(self.history['lr'])
        plt.xlabel('Training Iteration')
        plt.ylabel('Learning rate')
        plt.show()
        fig_acc.savefig(results_dir + "/model_cyclic_lr.png")

        plt.cla()
        plt.close()<jupyter_output><empty_output><jupyter_text>Utils<jupyter_code>def plot_metric(history_in, metric_name, results_dir):
    """
    Plot a metric of model's history.
    """

    fig_acc = plt.figure(figsize=(10, 10))
    plt.plot(history_in.history[metric_name])
    plt.plot(history_in.history['val_' + metric_name])

    plt.title('model ' + metric_name)
    plt.ylabel(metric_name)
    plt.xlabel('epoch')
    plt.legend(['train', 'validation'], loc='upper left')
    fig_acc.savefig(os.path.join(results_dir, "model_" + metric_name + ".png"))

    plt.cla()
    plt.close()


def mkdir_if_not_exist(directory):

        if not os.path.exists(directory):
            try:
                os.makedirs(directory)
            except OSError as e:
                print("Error: %s - %s." % (e.filename, e.strerror))<jupyter_output><empty_output><jupyter_text>Data Preparation<jupyter_code># Cloud TPUs can only access data in GCS as only the GCS file system is registered.
# https://stackoverflow.com/questions/62870656/file-system-scheme-local-not-implemented-in-google-colab-tpu
# Colab can't use KaggleDatasets library, so we need to save the GCS path by ourself
# copying it from a Kaggle notebook.
GCS_DS_PATH = "gs://kds-8e853601547c2d2e7a243638123e66bb32174c3a2b50da35aca5baed"
df = pd.read_csv("/content/train.csv")
print(f"{df.columns}")
print(len(df), df['StudyInstanceUID'].nunique())
print('Number of Records:',len(df), 'Number of Patients:' ,df['PatientID'].nunique())
paths = GCS_DS_PATH + "/train/" + df['StudyInstanceUID'] + '.jpg'
paths[0]
# Get the multi-labels
df_sub = pd.read_csv("/content/sample_submission.csv")
labels_cols = df_sub.columns[1:]
n_classes = len(labels_cols)
print(f"{n_classes} - {labels_cols}")
labels = df[labels_cols].values
labels
id_label_map = {k: v for k, v in zip(paths, labels)}
#@title Configuration
#@markdown 
train_limit = 30083  #@param {type: "number"}
test_limit = 3582  #@param {type: "number"}
img_size =  600#@param {type:"integer"}
batch_size_in =   8#@param {type: "integer", min: 8, max: 128}
#@markdown Disable it in case of using EfficientNets. The input data should range [0, 255]. Normalization is included as part of the model.
use_normalize = False #@param {type:"boolean"}<jupyter_output><empty_output><jupyter_text>Build train and validation datasets<jupyter_code>ndarr_paths = np.reshape(paths.to_numpy(), (paths.shape[0], 1))
ndarr_paths
batch_size = strategy.num_replicas_in_sync * batch_size_in
print(f"Batch size: {batch_size}")

# train_paths, valid_paths, train_labels, valid_labels = train_test_split(paths[0:train_limit], labels[0:train_limit], test_size=0.1, shuffle=True, random_state=SEED)

# Multi-label data stratification
train_paths, train_labels, valid_paths, valid_labels = iterative_train_test_split(ndarr_paths[0:train_limit], labels[0:train_limit], test_size = 0.1)
train_paths = train_paths.flatten()
valid_paths = valid_paths.flatten()

print(f"# Training images: {train_paths.shape}")
print(f"# Training labels: {train_labels.shape}")
print(f"# Validation images: {valid_paths.shape}")
print(f"# Validation labels: {valid_labels.shape}")
print(f"# Training steps: {len(train_paths) // batch_size}")
print(f"# Validation steps: {len(valid_paths) // batch_size}")
# Build the tensorflow datasets
decoder = build_decoder(target_size=(img_size, img_size), 
                        with_labels=True,
                        normalize=use_normalize)

# Build the tensorflow datasets
dtrain = build_dataset(
    train_paths, 
    train_labels, 
    bsize=batch_size, 
    decode_fn=decoder
)

dvalid = build_dataset(
    valid_paths, 
    valid_labels, 
    bsize=batch_size, 
    repeat=False, 
    shuffle=False, 
    augment=False, 
    decode_fn=decoder
)

dtrain
for image, label in dtrain.take(1):
  print(image[0].numpy())<jupyter_output>[[[ 4.255432   4.255432   4.255432 ]
  [ 3.95578    3.95578    3.95578  ]
  [ 5.655609   5.655609   5.655609 ]
  ...
  [ 3.700821   3.700821   3.700821 ]
  [ 3.95578    3.95578    3.95578  ]
  [ 2.7279968  2.7279968  2.7279968]]

 [[10.72805   10.72805   10.72805  ]
  [14.674362  14.674362  14.674362 ]
  [16.62915   16.62915   16.62915  ]
  ...
  [13.022209  13.022209  13.022209 ]
  [13.180283  13.180283  13.180283 ]
  [ 9.869186   9.869186   9.869186 ]]

 [[12.921448  12.921448  12.921448 ]
  [14.895325  14.895325  14.895325 ]
  [13.617691  13.617691  13.617691 ]
  ...
  [14.589371  14.589371  14.589371 ]
  [12.889633  12.889633  12.889633 ]
  [ 9.320168   9.320168   9.320168 ]]

 ...

 [[13.400055  13.400055  13.400055 ]
  [19.77359   19.77359   19.77359  ]
  [19.77359   19.77359   19.77359  ]
  ...
  [24.226906  24.226906  24.226906 ]
  [32.56414   32.56414   32.56414  ]
  [38.84468   38.84468   38.84468  ]]

 [[13.626175  13.626175  13.626175 ]
  [18.753746  18.753746  18.753746 ]
[...]<jupyter_text>Modeling<jupyter_code>#@title Configuration
#@markdown 
output_path = "model" #@param {type: "string"}
epochs = 30#@param {type:"integer"}
n_fold = 5  #@param {type: "slider", min: 2, max: 10}
architecture = "EfficientNetB3" #@param ["InceptionV3", "Xception","InceptionResNetV2","NASNetLarge","ResNet152V2", "EfficientNetB0", "EfficientNetB1", "EfficientNetB2", "EfficientNetB3", "EfficientNetB4", "EfficientNetB5", "EfficientNetB6", "EfficientNetB7"]
use_lr_strategy = False #@param {type:"boolean"}
use_lr_finder = True #@param {type:"boolean"}<jupyter_output><empty_output><jupyter_text>Single model
To get maxmimum accuracy, we leverage a pretrained image recognition model. We drop the ImageNet-specific top layers (`include_top=false`), and add a Global average pooling and a sigmoid layer to predict our 11 classes.
We also use [Focal loss](https://www.tensorflow.org/addons/api_docs/python/tfa/losses/SigmoidFocalCrossEntropy) since we have highly imbalanced classes. It down-weights well-classified examples and focuses on hard examples. The loss value is much high for a sample which is misclassified by the classifier as compared to the loss value corresponding to a well-classified example.  Find the best learning rate<jupyter_code>if use_lr_finder:
  model = compile_model(strategy, architecture, img_size, n_classes)
  lr_finder = LRFinder(min_lr=1e-5,
                      max_lr=1e-1,
                      steps_per_epoch=np.ceil(len(train_paths) // batch_size),
                      epochs=3)
  with strategy.scope():
    model.fit(dtrain, 
              batch_size=batch_size, 
              callbacks=[lr_finder], 
              steps_per_epoch=len(train_paths) // batch_size)

  lr_finder.plot_loss("")<jupyter_output>6/422 [..............................] - ETA: 3:30 - loss: 0.8867 - auc: 0.4717WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0074s vs `on_train_batch_end` time: 12.5083s). Check your callbacks.<jupyter_text>Train<jupyter_code>path = os.path.join("output_path", "single_model")
mkdir_if_not_exist(path)
model_path = os.path.join(path, f"{architecture}_model.h5")

schedule = SGDRScheduler(min_lr=1e-3,
                         max_lr=1e-1,
                         steps_per_epoch=len(train_paths) // batch_size,
                         lr_decay=0.8,
                         cycle_length=5,
                         mult_factor=1.5)

lr_strategy=schedule if use_lr_strategy else None

callbacks = get_callbacks(model_path, lr_strategy)

with strategy.scope():
  model = compile_model(strategy, architecture, img_size, n_classes)
  history = model.fit(
          dtrain,
          validation_data=dvalid,
          epochs=epochs, 
          verbose=1,
          callbacks=callbacks,
          steps_per_epoch=len(train_paths) // batch_size,
          validation_steps=len(valid_paths) // batch_size)

  # summarize history
  plot_metric(history, 'loss', path)
  plot_metric(history, 'auc', path)

# To avoid timeout 
print("Avoiding timeout...")
while True:pass
from google.colab import files
files.download(model_path)<jupyter_output><empty_output><jupyter_text>Predictions<jupyter_code>with strategy.scope():
    model = tf.keras.models.load_model(model_path)
test_decoder = build_decoder(with_labels=False, target_size=(img_size, img_size))

df_sub = pd.read_csv("/content/sample_submission.csv")
test_paths = GCS_DS_PATH + "/test/" + df_sub['StudyInstanceUID'] + '.jpg'

dtest = build_dataset(
    test_paths, bsize=batch_size, repeat=False, 
    shuffle=False, augment=False, cache=False, 
    decode_fn=test_decoder
)
dtest
y_preds = model.predict(dtest, verbose=1)
df_sub.iloc[:, 1:] = y_preds
display(df_sub)
df_sub.to_csv('submission.csv',index=False)<jupyter_output><empty_output><jupyter_text>TTA
A final technique that can raise accuracy by one percent or two is test time augmentation (TTA). This involves taking a series of different versions of the original image (for example cropping different areas, or changing the zoom) and passing them through the model. The average output is then calculated for the different versions and this is given as the final output score for the image.<jupyter_code># Test time augmentation rounds
TTA = 5
NUM_TESTING_IMAGES = df_sub.shape[0]
steps = TTA * ((NUM_TESTING_IMAGES / batch_size) + 1)

# Get the test dataset with tta to extract image
tta_dtest = build_tta(test_paths,
                      decode_fn=build_decoder(with_labels=False, 
                                              target_size=(img_size, img_size)),
                      augment_fn=build_augmenter(with_labels=False), 
                      bsize=batch_size, 
                      tta = True)

probabilities = model.predict(tta_dtest, steps = steps, verbose=1)[: TTA * NUM_TESTING_IMAGES]
probabilities = np.mean(probabilities.reshape((NUM_TESTING_IMAGES, TTA, n_classes), order = 'F'), axis = 1)

df_sub.iloc[:, 1:] = probabilities
display(df_sub)
df_sub.to_csv('submission_tta.csv',index=False)<jupyter_output><empty_output><jupyter_text>Ensemble  Train<jupyter_code>image_paths = paths[0:train_limit]
image_labels = labels[0:train_limit]

kf = IterativeStratification(n_splits=n_fold, order=1)

for fold in range(0, n_fold):

    print(f"\nFold: {fold}")

    path = os.path.join("output_path", str(fold))
    mkdir_if_not_exist(path)

    model_path = os.path.join(path, "model_"+str(fold)+".h5")

    result = next(kf.split(X=image_paths, y=image_labels), None)

    train = image_paths[result[0]].tolist()
    train_labels = [id_label_map[k] for k in train]
    val = image_paths[result[1]].tolist()
    valid_labels = [id_label_map[k] for k in val]

    print(f"train shape: {len(train)}")
    print(f"train steps_per_epoch: {len(train) // batch_size}")
    print(f"val shape: {len(val)}")
    print(f"validation steps_per_epoch: {len(val) // batch_size}")

    # Build the tensorflow datasets
    dtrain = build_dataset(
        train, train_labels, bsize=batch_size, decode_fn=decoder
    )

    dvalid = build_dataset(
        val, valid_labels, bsize=batch_size, 
        repeat=False, shuffle=False, augment=False, decode_fn=decoder
    )

    tf.keras.backend.clear_session()

    schedule = SGDRScheduler(min_lr=1e-3,
                         max_lr=1e-1,
                         steps_per_epoch=np.ceil(len(train)/batch_size),
                         lr_decay=0.8,
                         cycle_length=5,
                         mult_factor=1.5)

    lr_strategy=schedule if use_lr_strategy else None


    callbacks = get_callbacks(model_path, lr_strategy)

    with strategy.scope():
      model = compile_model(strategy, architecture, img_size, n_classes)
      
      ###########
      # Training
      ###########

      history = model.fit(
          dtrain,
          validation_data=dvalid,
          epochs=epochs, verbose=1,
          callbacks=callbacks,
          steps_per_epoch=len(train) // batch_size,
          validation_steps=len(val) // batch_size)

      # summarize history
      plot_metric(history, 'loss', path)
      plot_metric(history, 'auc', path)
      
      # in order to release the memory
      del history
      del model
      gc.collect()

from google.colab import files

for fold in range(0, n_fold):
  print(f"downloading model_{fold}...")
  path = os.path.join("output_path", str(fold))
  mkdir_if_not_exist(path)
  model_path = os.path.join(path, "model_"+str(fold)+".h5")
  files.download(model_path) 

# To avoid timeout 
print("Avoiding timeout...")
while True:pass<jupyter_output>Fold: 0
train shape: 24098
train steps_per_epoch: 376
val shape: 5985
validation steps_per_epoch: 93<jupyter_text>Predictions<jupyter_code>test_paths = GCS_DS_PATH + "test/" + df_sub['StudyInstanceUID'] + '.jpg'
test_paths[0]
test_decoder = build_decoder(with_labels=False, target_size=(img_size, img_size))

dtest = build_dataset(
    test_paths[0:test_limit], bsize=batch_size, repeat=False, 
    shuffle=False, augment=False, cache=False, 
    decode_fn=test_decoder
)

ensemble_preds = np.zeros((len(test_paths[0:test_limit]), n_classes), dtype=np.float)

for fold in range(0, n_fold):
  print(f"model: {fold}")
  path = os.path.join("output_path", str(fold))
  mkdir_if_not_exist(path)
  model_path = os.path.join(path, "model_"+str(fold)+".h5")
  
  with strategy.scope():
      model = tf.keras.models.load_model(model_path)

      y_preds = model.predict(dtest, verbose=1)

      # sum the predicted values
      ensemble_preds += np.array(y_preds, dtype=np.float)

# average the predicted values
ensemble_preds /= n_fold
df_sub.iloc[:, 1:] = ensemble_preds
display(df_sub)<jupyter_output>model: 0
56/56 [==============================] - 82s 1s/step
model: 1
56/56 [==============================] - 81s 1s/step
model: 2
56/56 [==============================] - 83s 1s/step
model: 3
56/56 [==============================] - 84s 1s/step
model: 4
56/56 [==============================] - 84s 1s/step
{'fraction_non_alphanumeric': 0.11571114138979142, 'fraction_numerical': 0.035085420815428155, 'mean_word_length': 3.2752324409096003, 'pattern_counts': {'":': 13, '<': 80, '<?xml version=': 0, '>': 83, 'https://': 20, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 8, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 25, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '619430', 'license': 'Apache-2.0', 'n_tokens_mistral': 13756, 'n_tokens_neox': 12579, 'n_words': 2681, 'path': 'baseline.ipynb', 'repo_name': 'umbertogriffo/kaggle-ranzcr-catheter-and-line-position-challenge-baseline', 'size': '38185'}