<jupyter_start><jupyter_text>The Data Generators<jupyter_code># Defining constants:
BATCH_SIZE = 32
IMAGE_SIZE = 224
CLASSES = [ 
  'Atelectasis',
  'Cardiomegaly',
  'Consolidation',
  'Edema',
  'Pleural Effusion'
]<jupyter_output><empty_output><jupyter_text>Generating a sample for datagen fitting for featurewise normalization and featurewise centering.<jupyter_code># Constants
FRAC = 0.003 # Fraction of total data to be taken as sample
SHAPE = (320, 390, 3) # Common shape for featurewise centering & normalization

sample_paths = df['Path'].sample(frac=FRAC).to_numpy()
X_temp = np.array([np.array(cv2.imread(path, 1), dtype=float) for path in sample_paths])
X_sample = np.array([x for x in X_temp if x.shape == SHAPE])
from keras.preprocessing.image import ImageDataGenerator as IDG

datagen = IDG(
    rescale=1./255, 
    featurewise_center=True,
    featurewise_std_normalization=True,
    rotation_range=0.1,
    zoom_range = 0.1,
    width_shift_range=0.2,
    height_shift_range=0.2,
    horizontal_flip=True,
    validation_split = 0.1,
    fill_mode = 'nearest'
)

datagen.fit(X_sample)

test_datagen = IDG(rescale=1./255)
def get_gen():
  train_gen = datagen.flow_from_dataframe(
      dataframe = df,
      #directory = '/content/CheXpert-v1.0-small/train',
      x_col = 'Path',
      y_col = CLASSES, #'classes',
      class_mode='raw',
      #validate_filenames = False,
      seed=42,
      shuffle=True,
      target_size=(IMAGE_SIZE, IMAGE_SIZE), 
      batch_size=BATCH_SIZE, 
      subset = 'training'
  )

  val_gen = datagen.flow_from_dataframe(
      dataframe = df,
      #directory = '/content/CheXpert-v1.0-small/train',
      x_col = 'Path',
      y_col = CLASSES, #'classes',
      class_mode='raw',
      #validate_filenames = False,
      seed=42,
      shuffle=True,
      target_size=(IMAGE_SIZE, IMAGE_SIZE), 
      batch_size=BATCH_SIZE, 
      #classes = columns,
      subset = 'validation'
  )

  return train_gen, val_gen<jupyter_output><empty_output><jupyter_text>Cyclic LRIt seems that our code gets stuck on a loss plateau. Let's try to implement the cyclic LR callback. The below callback implements a cyclical learning rate policy (CLR).The method cycles the learning rate between two boundaries withsome constant frequency, as detailed in this paper (https://arxiv.org/abs/1506.01186).The amplitude of the cycle can be scaled on a per-iteration or per-cycle basis.This class has three built-in policies, as put forth in the paper.    "triangular":        A basic triangular cycle w/ no amplitude scaling.    "triangular2":        A basic triangular cycle that scales initial amplitude by half each cycle.    "exp_range":        A cycle that scales initial amplitude by gamma**(cycle iterations) at each         cycle iteration.    For more detail, please see paper.         Example        ```python            clr = CyclicLR(base_lr=0.001, max_lr=0.006,                                step_size=2000., mode='triangular')            model.fit(X_train, Y_train, callbacks=[clr])        ```        Class also supports custom scaling functions:        ```python            clr_fn = lambda x: 0.5*(1+np.sin(x*np.pi/2.))            clr = CyclicLR(base_lr=0.001, max_lr=0.006,                                step_size=2000., scale_fn=clr_fn,                                scale_mode='cycle')            model.fit(X_train, Y_train, callbacks=[clr])        ```         base_lr: initial learning rate which is the            lower boundary in the cycle.        max_lr: upper boundary in the cycle. Functionally,            it defines the cycle amplitude (max_lr - base_lr).            The lr at any cycle is the sum of base_lr            and some scaling of the amplitude; therefore             max_lr may not actually be reached depending on            scaling function.        step_size: number of training iterations per            half cycle. Authors suggest setting step_size            2-8 x training iterations in epoch.        mode: one of {triangular, triangular2, exp_range}.            Default 'triangular'.            Values correspond to policies detailed above.            If scale_fn is not None, this argument is ignored.        gamma: constant in 'exp_range' scaling function:            gamma**(cycle iterations)        scale_fn: Custom scaling policy defined by a single            argument lambda function, where             0 = 0.            mode paramater is ignored         scale_mode: {'cycle', 'iterations'}.            Defines whether scale_fn is evaluated on             cycle number or cycle iterations (training            iterations since start of cycle). Default is 'cycle'.<jupyter_code>from keras.callbacks import *

class CyclicLR(Callback):
    
    def __init__(self, base_lr=0.001, max_lr=0.01, step_size=2000., mode='triangular',
                 gamma=1., scale_fn=None, scale_mode='cycle'):
        super(CyclicLR, self).__init__()

        self.base_lr = base_lr
        self.max_lr = max_lr
        self.step_size = step_size
        self.mode = mode
        self.gamma = gamma
        if scale_fn == None:
            if self.mode == 'triangular':
                self.scale_fn = lambda x: 1.
                self.scale_mode = 'cycle'
            elif self.mode == 'triangular2':
                self.scale_fn = lambda x: 1/(2.**(x-1))
                self.scale_mode = 'cycle'
            elif self.mode == 'exp_range':
                self.scale_fn = lambda x: gamma**(x)
                self.scale_mode = 'iterations'
        else:
            self.scale_fn = scale_fn
            self.scale_mode = scale_mode
        self.clr_iterations = 0.
        self.trn_iterations = 0.
        self.history = {}

        self._reset()

    def _reset(self, new_base_lr=None, new_max_lr=None,
               new_step_size=None):
        """Resets cycle iterations.
        Optional boundary/step size adjustment.
        """
        if new_base_lr != None:
            self.base_lr = new_base_lr
        if new_max_lr != None:
            self.max_lr = new_max_lr
        if new_step_size != None:
            self.step_size = new_step_size
        self.clr_iterations = 0.
        
    def clr(self):
        cycle = np.floor(1+self.clr_iterations/(2*self.step_size))
        x = np.abs(self.clr_iterations/self.step_size - 2*cycle + 1)
        if self.scale_mode == 'cycle':
            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(cycle)
        else:
            return self.base_lr + (self.max_lr-self.base_lr)*np.maximum(0, (1-x))*self.scale_fn(self.clr_iterations)
        
    def on_train_begin(self, logs={}):
        logs = logs or {}

        if self.clr_iterations == 0:
            K.set_value(self.model.optimizer.lr, self.base_lr)
        else:
            K.set_value(self.model.optimizer.lr, self.clr())        
            
    def on_batch_end(self, epoch, logs=None):
        
        logs = logs or {}
        self.trn_iterations += 1
        self.clr_iterations += 1

        self.history.setdefault('lr', []).append(K.get_value(self.model.optimizer.lr))
        self.history.setdefault('iterations', []).append(self.trn_iterations)

        for k, v in logs.items():
            self.history.setdefault(k, []).append(v)
        
        K.set_value(self.model.optimizer.lr, self.clr())<jupyter_output><empty_output><jupyter_text>The Model<jupyter_code># Building on top of the base:
from keras.applications import Xception
from keras.models import Sequential
from keras.layers import BatchNormalization, Conv2D, GlobalAveragePooling2D
from keras.layers.core import Flatten, Dense, Dropout

def build_model():
  # The convolutional base:
  model_base = Xception(
      weights='imagenet', include_top=False, input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3)
      )
  #model_base.trainable = False
  # Unfreezing all the layers:
  for layer in model_base.layers:
      layer.trainable = True

  model = Sequential()
  model.add(model_base) # Adding the base as a layer
  model.add(GlobalAveragePooling2D())
  model.add(Dense(1024, activation='relu'))
  model.add(BatchNormalization())
  model.add(Dropout(0.3))
  #model.add(Flatten())
  #model.add(Dense(1024, activation='relu'))
  #model.add(Dropout(0.25))
  model.add(Dense(5, activation='sigmoid'))
  
  return model<jupyter_output><empty_output><jupyter_text>Training and Validating<jupyter_code>from keras.metrics import AUC, categorical_accuracy as catacc
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras.optimizers import Adam

auc = AUC()
adam = Adam(learning_rate=0.00005) # 0.05 of default

es = EarlyStopping(monitor='val_categorical_accuracy', mode='max', verbose=1, patience=2)
mc = ModelCheckpoint(
    filepath='xception-keras-2.h5', verbose=1 #, save_best_only=True
)

clr = CyclicLR(base_lr=0.00005, max_lr=0.0001)
cb_list = [es, mc] # Will add clr later, as we'll have to tune it's hyperparameters

model = build_model()
model.compile(
    loss='binary_crossentropy',
    optimizer=adam,
    metrics=[auc, catacc] # Earlier, 'acc' 
)
train_gen, val_gen = get_gen()
# Training constants:
TRAIN_STEPS = train_gen.n//BATCH_SIZE
VAL_STEPS   = val_gen.n//BATCH_SIZE
N_EPOCHS = 5
history = model.fit_generator(
    train_gen,
    steps_per_epoch=TRAIN_STEPS,
    epochs=N_EPOCHS,
    validation_data=val_gen,
    validation_steps=VAL_STEPS,
    callbacks = cb_list
)
cat_acc = history.history['categorical_accuracy']
val_cat_acc = history.history['val_categorical_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(cat_acc) + 1)

plt.plot(epochs, cat_acc, 'b', label='Training accuracy')
plt.plot(epochs, val_cat_acc, 'r', label='Validation accuracy')
plt.title('Training and validation accuracy')
plt.legend()
plt.figure()

plt.plot(epochs, loss, 'b', label='Training loss')
plt.plot(epochs, val_loss, 'r', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Test Generator<jupyter_code>test_gen = test_datagen.flow_from_dataframe(
    dataframe = df_val,
    #directory = '/content/CheXpert-v1.0-small/valid',
    x_col = 'Path',
    y_col = CLASSES, #'classes',
    class_mode='raw',
    #validate_filenames = False,
    target_size=(IMAGE_SIZE, IMAGE_SIZE), 
    batch_size=1, 
    shuffle = False,
    #classes = columns,
)<jupyter_output>Found 234 validated image filenames.<jupyter_text>ROC-AUC Curves<jupyter_code>#from keras.models import load_model
#mod = load_model('xception-keras-2.h5')
y_labels = df_val[CLASSES].to_numpy()
y_pred = model.predict_generator(test_gen, steps=test_gen.n) 
# As batch size for this is kept 1, steps = len(test_gen)
from sklearn.metrics import roc_curve
from sklearn.metrics import auc

plt.figure(1)
plt.plot([0, 1], [0, 1], 'k--')

for i in range(len(CLASSES)):
   fpr, tpr, thresholds = roc_curve(y_labels[:, i], y_pred[:, i])
   individual_auc = auc(fpr, tpr)
   plt.plot(fpr, tpr, label= (CLASSES[i] + '(area = {})'.format(individual_auc)))

    
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.title('ROC curve')
plt.legend(loc='best')
plt.show()
results = model.evaluate_generator(test_gen)
results<jupyter_output><empty_output><jupyter_text>Stuff to try out:- Optimize the initial LR- Reduce LR on Plateau- Cyclic learning rates  Classifier Architectures:- A1: AUC: (76, 71) | cat_acc: (32, 30)```model = Sequential()  model.add(model_base)  Adding the base as a layer  model.add(Conv2D(500, kernel_size = (3,3), activation='relu', input_shape = (IMAGE_SIZE,IMAGE_SIZE,3)))  model.add(BatchNormalization())  model.add(Dropout(0.2))  model.add(GlobalAveragePooling2D())  model.add(Dense(1024, activation='relu'))  model.add(BatchNormalization())  model.add(Dropout(0.2))  model.add(Dense(5, activation='sigmoid'))  ```    - A2: AUC: (76, 69) | cat_acc: (31, 42)```model.add(GlobalAveragePooling2D())model.add(Dense(5, activation='sigmoid'))```  - A3: AUC: (87, 86) | cat_acc: (46, 45) | lr=$0.00005$```model.add(GlobalAveragePooling2D())model.add(Dense(1024, activation='relu'))model.add(BatchNormalization())model.add(Dropout(0.3))model.add(Dense(5, activation='sigmoid'))```<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11733943454812878, 'fraction_numerical': 0.01944762216048374, 'mean_word_length': 3.2422876949740034, 'pattern_counts': {'":': 3, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '515991', 'license': 'MIT', 'n_tokens_mistral': 4251, 'n_tokens_neox': 3992, 'n_words': 935, 'path': 'Xception/notebook_xception.ipynb', 'repo_name': 'PerXeptron/X-Ray-Anomaly-Detection-Models', 'size': '12238'}