<jupyter_start><jupyter_text>To get started you need to setup a Python environment with NumPy, Matplotlib and TensorFlow 2.0. To use the PatchCAMELYON dataset with TensorFlow Datasets you will need to use my fork of the project for now as the pull request to add PatchCAMELYON to the master branch is not yet approved. To this end you need to do clone the repository and add it to your Python environment:<jupyter_code># Import NumPy to handle array's and Matplotlib for plotting loss curves
import numpy as np
import matplotlib.pyplot as plt

# Import TensorFlow and relevant Keras classes to setup the model
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Conv2D, MaxPool2D, Flatten, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.callbacks import ModelCheckpoint<jupyter_output><empty_output><jupyter_text>The next cell will automatically download PatchCAMELYON from Zenodo and prepare the TensorFlow Datasets<jupyter_code>import tensorflow_datasets as tfds
pcam, pcam_info = tfds.load("patch_camelyon", with_info=True)
print(pcam_info)<jupyter_output>[1mDownloading and preparing dataset patch_camelyon/2.0.0 (download: 7.48 GiB, generated: Unknown size, total: 7.48 GiB) to /root/tensorflow_datasets/patch_camelyon/2.0.0...[0m<jupyter_text>After this step you should be able to import the relevant packages with the following cell<jupyter_code>input_img = Input(shape=(96,96,3))

# Now we define the layers of the convolutional network: three blocks of two convolutional layers and a max-pool layer.
x = Conv2D(16, (3, 3), padding='valid', activation='relu')(input_img)
x = Conv2D(16, (3, 3), padding='valid', activation='relu')(x)
x = MaxPool2D(pool_size=(2,2), strides=(2,2))(x)
x = Conv2D(32, (3, 3), padding='valid', activation='relu')(x)
x = Conv2D(32, (3, 3), padding='valid', activation='relu')(x)
x = MaxPool2D(pool_size=(2,2), strides=(2,2))(x)
x = Conv2D(64, (3, 3), padding='valid', activation='relu')(x)
x = Conv2D(64, (3, 3), padding='valid', activation='relu')(x)
x = MaxPool2D(pool_size=(2,2), strides=(2,2))(x)

# Now we flatten the output from a 4D to a 2D tensor to be able to use fully-connected (dense) layers for the final
# classification part. Here we also use a bit of dropout for regularization. The last layer uses a softmax to obtain class
# likelihoods (i.e. metastasis vs. non-metastasis)
x = Flatten()(x)
x = Dense(256, activation='relu')(x)
x = Dropout(rate=0.2)(x)
x = Dense(128, activation='relu')(x)
x = Dropout(rate=0.2)(x)
predictions = Dense(2, activation='softmax')(x)

# Now we define the inputs/outputs of the model and setup the optimizer. In this case we use regular stochastic gradient
# descent with Nesterov momentum. The loss we use is cross-entropy and we would like to output accuracy as an additional metric.
model = Model(inputs=input_img, outputs=predictions)
sgd_opt = SGD(lr=0.01, momentum=0.9, decay=0.0, nesterov=True)
model.compile(optimizer=sgd_opt,
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.summary()<jupyter_output>Model: "functional_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 96, 96, 3)]       0         
_________________________________________________________________
conv2d (Conv2D)              (None, 94, 94, 16)        448       
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 92, 92, 16)        2320      
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 46, 46, 16)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 44, 44, 32)        4640      
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 42, 42, 32)        9248      
______________________________________________________[...]<jupyter_text>Model: “model”Layer (type) Output Shape Param input_1 (InputLayer) [(None, 96, 96, 3)] 0conv2d (Conv2D) (None, 94, 94, 16) 448conv2d_1 (Conv2D) (None, 92, 92, 16) 2320max_pooling2d (MaxPooling2D) (None, 46, 46, 16) 0conv2d_2 (Conv2D) (None, 44, 44, 32) 4640conv2d_3 (Conv2D) (None, 42, 42, 32) 9248max_pooling2d_1 (MaxPooling2 (None, 21, 21, 32) 0conv2d_4 (Conv2D) (None, 19, 19, 64) 18496conv2d_5 (Conv2D) (None, 17, 17, 64) 36928max_pooling2d_2 (MaxPooling2 (None, 8, 8, 64) 0flatten (Flatten) (None, 4096) 0dense (Dense) (None, 256) 1048832dropout (Dropout) (None, 256) 0dense_1 (Dense) (None, 128) 32896dropout_1 (Dropout) (None, 128) 0dense_2 (Dense) (None, 2) 258Total params: 1,154,066 Trainable params: 1,154,066 Non-trainable params: 0<jupyter_code>def convert_sample(sample):
    image, label = sample['image'], sample['label']  
    image = tf.image.convert_image_dtype(image, tf.float32)
    label = tf.one_hot(label, 2, dtype=tf.float32)
    return image, label<jupyter_output><empty_output><jupyter_text>Now we use the tf.data pipeline to apply this function to the dataset in a parallel fashion. We also shuffle the training data with a shuffle buffer (which is randomly filled with samples from the dataset) of 1024. Next we define batches of 64 patches for training and 128 for validation. Last, we prefetch 2 batches such that we can get batches during training on the GPU.<jupyter_code>train_pipeline = pcam['train'].map(convert_sample,
                                   num_parallel_calls=8).shuffle(1024).repeat().batch(64).prefetch(2)
valid_pipeline = pcam['validation'].map(convert_sample,
                                        num_parallel_calls=8).repeat().batch(128).prefetch(2)<jupyter_output><empty_output><jupyter_text>Now we just apply train and evaluate the model using our dataset pipeline. We pick the steps per epoch such that the entire training and validation set are covered each epoch. We keep the History object that fit returns to plot the loss progression later on. We now just do 5 epochs for illustration purposes. Feel free to experiment with the number of epochs. If you want to keep the best model during training you can use the Keras ModelCheckpoint callback to write each improvement to disk.<jupyter_code>hist = model.fit(train_pipeline,
                 validation_data=valid_pipeline,
                 verbose=2, epochs=5, steps_per_epoch=4096, validation_steps=256)<jupyter_output>Epoch 1/5
WARNING:tensorflow:Callbacks method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0169s vs `on_train_batch_end` time: 0.0301s). Check your callbacks.<jupyter_text>Epoch 1/5 4096/4096 - 101s - loss: 0.6756 - accuracy: 0.5501 - val_loss: 0.5228 - val_accuracy: 0.7527 Epoch 2/5 4096/4096 - 98s - loss: 0.4292 - accuracy: 0.8071 - val_loss: 0.3946 - val_accuracy: 0.8249 Epoch 3/5 4096/4096 - 99s - loss: 0.3165 - accuracy: 0.8675 - val_loss: 0.3634 - val_accuracy: 0.8390 Epoch 4/5 4096/4096 - 100s - loss: 0.2586 - accuracy: 0.8958 - val_loss: 0.3707 - val_accuracy: 0.8340 Epoch 5/5 4096/4096 - 99s - loss: 0.2260 - accuracy: 0.9118 - val_loss: 0.3353 - val_accuracy: 0.8568<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10742346247629368, 'fraction_numerical': 0.07206719046328908, 'mean_word_length': 4.687981510015408, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '224587', 'license': 'CC0-1.0', 'n_tokens_mistral': 2673, 'n_tokens_neox': 2304, 'n_words': 764, 'path': 'PatchCam.ipynb', 'repo_name': 'CharlesSS07/DeepLearningCamelyon', 'size': '7382'}