<jupyter_start><jupyter_text>Detect distracted drivers*<NAME> - June 1st 2018***This jupyter notebook is a presentation of my work. For a more precise code with MLFlow tracking, please see `src/train.py`.** The goal is to predict what the driver is doing based on picturesThis project is from a [Kaggle competition : State Farm Distracted Driver Detection](https://www.kaggle.com/c/state-farm-distracted-driver-detection)There are 10 classes to predict:* c0: safe driving* c1: texting - right* c2: talking on the phone - right* c3: texting - left* c4: talking on the phone - left* c5: operating the radio* c6: drinking* c7: reaching behind* c8: hair and makeup* c9: talking to passenger Explanation of my workI used a pre-trained VGG-16 Convolutional Neural Network as a base layer. I then removed the last layer (=top layer) and added a Dense layer with a softmax to output the classification. The optimization algorithm is Adam with a small learning rate: 0.0001. The used the weights trained VGG-16 on the image-net dataset for the initializatzion. All the layers beside the last 3 layers were frozen (last 3 layers are fully connected layers). I then performed data augmentation because the training set is not big. I fine tuned the model using the training set and a validation set (30% split)I used the Keras library as much as possible in this project as it allows to prototype quickly and surely. I used the dataflow and datagen functions to load the file and preprocess them while feeding them by batches directly to the model.4 epochs only were used. The learning_rate, number of epochs and numbers of layers to freeze values were determined by several experiments, not shown here. There is an early stopping to adjust the number of epochs.The framework was run on a Google Cloud Virtual Machine with a Tesla K80 GPU. Result:* Public Leaderboard: 1.039 (top 35%)* Private Leaderboard 0.86562 (top 30%) To predict:* Run the last cells of this notebook* Or use the script `predict.py [Model/model_name]` with the desired model to obtain prediction and generate a .csv file for Kaggle. Content:1. Model building2. Preprocessing and data augmentation3. Train4. Predict  Config to adjust:<jupyter_code># Params 
batch_size = 144 # Try 48; 72; 109; 144; 218; 327 depending on the computing power
n_epoch = 4
learning_rate = 0.0001
n_layers_train = 3 # last n_layers_train layers will be trained (in VGG16, value 3 to train last FC layers)
data_augmentation = 0 # True to augment image by transformation in the training phase
import pandas as pd
import cv2
import os 
import glob
import gc
from joblib import Parallel, delayed

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpli
% matplotlib inline
import warnings
warnings.filterwarnings("ignore", module="matplotlib")
pd.options.mode.chained_assignment = None

from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss

from keras.applications import vgg16
from keras.preprocessing import image
from keras import optimizers, callbacks
from keras.models import Sequential, load_model, Model
from keras.layers import Flatten, Dense 
from keras.utils import to_categorical
from tensorflow.python.client import device_lib
import tensorflow as tf

from livelossplot import PlotLossesKeras
import mlflow<jupyter_output>Using TensorFlow backend.<jupyter_text>First we can observe the distribution of samples per class. This is important as it is a classification problem. In order to avoid the [accuracy paradox](https://en.wikipedia.org/wiki/Accuracy_paradox), we don't want to have classes too imbalanced. Here the classes are pretty well balanced. All good !<jupyter_code>img_list = pd.read_csv('Data/driver_imgs_list.csv')

img_list['class_type'] = img_list['classname'].str.extract('(\d)',expand=False).astype(np.int)
plt.figure()
img_list.groupby(['class_type'])['subject'].count().plot(kind='bar',alpha=0.9, layout=(1,1), color='b')
plt.title('class distribution')
plt.xticks(range(0,11))
plt.xlabel('Class')
plt.show()<jupyter_output><empty_output><jupyter_text>1. Data loading and preprocessingTo load the files quickly, I use a multi-processor approach which allows to multiply by at least 5 the loading speed.The data is split in a train/validation set. Then I can use data augmentation with transformation to increase the size of the training set (not used here).<jupyter_code>def preprocess_image(img, mode='vgg16'):
    """Preprocess an image according to VGG16 imagenet requirement (mean substraction)
    Args:
    - img: image in BGR format with shape: [w,h,channel]
    - mode: type of preprocessing wanted according to model requirements
    Return:
    - img: preprocessed image
    """ 
    if(mode=='vgg16'):
        img = img.astype(np.float16)
        img[:, :, 0] -= 103.939
        img[:, :, 1] -= 116.779
        img[:, :, 2] -= 123.68
    return img

def load_image(img_file, img_reshape_size):
    """Load an image
    Args:
    - img_file: image file
    - img_reshape_size: shape(w,h) to resize the image
    Return:
    - img: image resized and preprocessed according to VGG16 standards
    """   
    img = cv2.imread(img_file)
    img = cv2.resize(img, img_reshape_size)
    img = preprocess_image(img)
    
    return img

def load_train_dataset(dataset_dir, img_reshape_size, nprocs=10):
    """Load the images located in the main folder dataset_dir Each class is in a separate subfolder
    Args:
    - dataset_dir: path to the directory containing subdirectories of images
    - img_reshape_size: shape(w,h) to resize the image
    - nprocs:Number of processors to use
    Return:
    - X: numpy array with each image data as a row
    - y: numpy array with each class as an integer for each image
    """
    X = []
    y = []
    # Train dataset
    for i in range(10):
        path = os.path.join(dataset_dir, 'c'+str(i),'*.jpg')
        files = glob.glob(path)

        X.extend(Parallel(n_jobs=nprocs)(delayed(load_image)(im_file, img_reshape_size) for im_file in files))
        y.extend([i]*len(files))
        print('folder train/c'+str(i), 'loaded')

    X = np.asarray(X, dtype=np.float16)
    y = np.asarray(y)
    return X, y
# Image sizes - Requirements of the CNN model
img_reshape_size = (224,224)

# Working directories
dataset_dir = 'Data'
dataset_dir_train = os.path.join(dataset_dir, 'train')

# Load train dataset
print('Load dataset train')
X_train_, y_train_ = load_train_dataset(dataset_dir_train, img_reshape_size) 
        
# Creation of a validation set
X_train, X_val, y_train, y_val = train_test_split(X_train_, y_train_, 
                                                  test_size=0.3, 
                                                  random_state=35)

# One hot encoding of the classes
y_train = to_categorical(y_train)
y_val = to_categorical(y_val)

# Shapes 
print('X_train shape:', X_train.shape)
print('y_train shape:', y_train.shape)#
print('X_val shape:', X_val.shape)
print('y_val shape:', y_val.shape)

# Image Data Generator for preprocessing and data augmentation
if(data_augmentation==True):
    train_datagen = image.ImageDataGenerator(preprocessing_function=preprocess_input_image,
                                            width_shift_range=0.1,
                                            height_shift_range=0.1,
                                            shear_range=0.1,
                                            rotation_range=8,
                                            fill_mode='nearest'
                                            )
else:
    train_datagen = image.ImageDataGenerator()
    
train_generator = train_datagen.flow(x=X_train, y=y_train,
                            batch_size=batch_size,
                            shuffle=False,
                            seed=42)<jupyter_output>Load dataset train
folder train/c0 loaded
folder train/c1 loaded
folder train/c2 loaded
folder train/c3 loaded
folder train/c4 loaded
folder train/c5 loaded
folder train/c6 loaded
folder train/c7 loaded
folder train/c8 loaded
folder train/c9 loaded
X_train shape: (15696, 224, 224, 3)
y_train shape: (15696, 10)
X_val shape: (6728, 224, 224, 3)
y_val shape: (6728, 10)<jupyter_text>2. Model buildingI use the VGG16 implementation and I remove the last layer to add a layer that suits my needs. In this case it is a 10 Fully connected layer so that I can use the CNN as a 10-class classifier.<jupyter_code>def create_VGG16_model(n_classes=10, n_layers_train=2, learning_rate=0.0001):
    """Load the images located in the main folder dataset_dir Each class is in a separate subfolder
    Args:
    - n_classes: number of classes to predict for the classifier
    - n_layers_train: number of last layers to train. The other layers will be frozen.
    Return:
    - model: Keras model
    """
    #Load the VGG model
    vgg16_base = vgg16.VGG16(weights='imagenet', include_top=True, input_shape=(224,224,3))    

    # Remove top layer and connect custom dense layer
    fc2 = vgg16_base.get_layer('fc2').output
    mypredictions = Dense(n_classes, activation='softmax', name='mypredictions')(fc2)
    model = Model(inputs=vgg16_base.input, outputs=mypredictions)
    
    model.summary()
    # Freeze the layers except the last n_layers_train layers
    for layer in model.layers[:-n_layers_train]:
        layer.trainable = False
    for layer in model.layers[:]:
        print(layer, layer.trainable)    


    model.compile(loss='categorical_crossentropy',
          optimizer=optimizers.Adam(lr=learning_rate),
          metrics=['accuracy'])

    model.summary()
    return model
model = create_VGG16_model(n_layers_train=n_layers_train, learning_rate=learning_rate)<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 224, 224, 3)       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 224, 224, 64)      1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 224, 224, 64)      36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 112, 112, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 112, 112, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 112, 112, 128)     147584    
_________________________________________________________________
block2_poo[...]<jupyter_text>3. Train<jupyter_code>#Callbacks:
# Tensorboard
tensorboard_log_dir='VGG16_lr'+str(learning_rate)+'_train'+str(n_layers_train)+'_epochs'+str(n_epoch)+'_data_aug'+str(data_augmentation)
tb_callback = callbacks.TensorBoard(log_dir=os.path.join('Graph', tensorboard_log_dir),
                                       histogram_freq=0, 
                                       write_graph=True, 
                                       write_images=False)
# Early stopping
earlystop = callbacks.EarlyStopping(monitor='val_loss', patience=2, \
                          verbose=1, mode='min')

# Fit the model on batches with real-time data augmentation:
model_history = model.fit_generator(train_generator,
                                    validation_data=(X_val, y_val),
                                    shuffle=True,
                                    epochs=n_epoch,
                                    steps_per_epoch=np.ceil(X_train.shape[0]//batch_size),
                                    callbacks=[tb_callback, earlystop],
                                    verbose=1,
                                    use_multiprocessing=True)<jupyter_output>Epoch 1/4
109/109 [==============================] - 264s 2s/step - loss: 0.3342 - acc: 0.9031 - val_loss: 0.0391 - val_acc: 0.9890
Epoch 2/4
109/109 [==============================] - 249s 2s/step - loss: 0.0129 - acc: 0.9971 - val_loss: 0.0273 - val_acc: 0.9926
Epoch 3/4
109/109 [==============================] - 251s 2s/step - loss: 0.0119 - acc: 0.9966 - val_loss: 0.0177 - val_acc: 0.9952
Epoch 4/4
109/109 [==============================] - 249s 2s/step - loss: 0.0075 - acc: 0.9976 - val_loss: 0.0245 - val_acc: 0.9935<jupyter_text>Now that the model is trained, we can save it for further predictions.Use the script `predict.py [Model/model_name]` to obtain predictions and generate a .csv file for Kaggle<jupyter_code># Save model and Exit
if not os.path.isdir('Model'):
    os.mkdir('Model')
filename = 'VGG16_lr'+str(learning_rate)+'_train'+str(n_layers_train)+'_epochs'+str(n_epoch)+'_data_aug'+str(data_augmentation)+'.h5'

model_file = os.path.join('Model', filename)
model.save(model_file)
print('File', filename, 'saved')<jupyter_output>File VGG16_lr0.0001_train3_epochs4_data_aug0.h5 saved<jupyter_text>4. Predict and submitUsing the model that was trained in this notebook, the script is going to predict the results for the test set and directly submit the results to Kaggle<jupyter_code>model_file = 'Model/VGG16_lr0.0001_train3_epochs4_data_aug0.h5'
%run -i 'src/predict.py' $model_file

!kaggle competitions submit -c state-farm-distracted-driver-detection -f "$(ls -t Output/submission-* | head -1)" -m "."<jupyter_output>Successfully submitted to State Farm Distracted Driver Detection
{'fraction_non_alphanumeric': 0.09709962168978563, 'fraction_numerical': 0.03597655960240338, 'mean_word_length': 3.4073226544622424, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '316053', 'license': 'MIT', 'n_tokens_mistral': 4250, 'n_tokens_neox': 3864, 'n_words': 1274, 'path': 'main_training.ipynb', 'repo_name': 'zhanglei1949/DistractedDriverDetection', 'size': '13489'}