<jupyter_start><jupyter_text>Building an image classification model using very little data  Based on the tutorial by <NAME> @fchollet https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html and the workbook by <NAME> https://github.com/gggdominici/keras-workshopThis tutorial presents several ways to build an image classifier using keras from just a few hundred or thousand pictures from each class you want to be able to recognize.We will go over the following options:  - training a small network from scratch (as a baseline)  - using the bottleneck features of a pre-trained network  - fine-tuning the top layers of a pre-trained network    This will lead us to cover the following Keras features:     - fit_generator for training Keras a model using Python data generators  - ImageDataGenerator for real-time data augmentation  - layer freezing and model fine-tuning  - ...and more.    Data Data can be downloaded at:https://www.kaggle.com/c/dogs-vs-cats/data  All you need is the train set  The recommended folder structure is:    Folder structure ```pythondata/    train/        dogs/  1024 pictures            dog001.jpg            dog002.jpg            ...        cats/  1024 pictures            cat001.jpg            cat002.jpg            ...    validation/        dogs/  416 pictures            dog001.jpg            dog002.jpg            ...        cats/  416 pictures            cat001.jpg            cat002.jpg            ...```Note : for this example we only consider 2x1000 training images and 2x400 testing images among the 2x12500 available. The github repo includes about 1500 images for this model. The original Kaggle dataset is much larger. The purpose of this demo is to show how you can build models with smaller size datasets. You should be able to improve this model by using more data.  Data loading<jupyter_code>##This notebook is built around using tensorflow as the backend for keras
!pip install pillow
!KERAS_BACKEND=tensorflow python -c "from keras import backend"
import os
import numpy as np
from keras.models import Sequential
from keras.layers import Activation, Dropout, Flatten, Dense
from keras.preprocessing.image import ImageDataGenerator
from keras.layers import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras import optimizers
# dimensions of our images.
img_width, img_height = 150, 150

train_data_dir = 'data/train'
validation_data_dir = 'data/validation'<jupyter_output><empty_output><jupyter_text>Imports<jupyter_code># used to rescale the pixel values from [0, 255] to [0, 1] interval
datagen = ImageDataGenerator(rescale=1./255)

# automagically retrieve images and their classes for train and validation sets
train_generator = datagen.flow_from_directory(
        train_data_dir,
        target_size=(img_width, img_height),
        batch_size=16,
        class_mode='binary')

validation_generator = datagen.flow_from_directory(
        validation_data_dir,
        target_size=(img_width, img_height),
        batch_size=32,
        class_mode='binary')<jupyter_output>Found 2048 images belonging to 2 classes.
Found 832 images belonging to 2 classes.<jupyter_text>Small Conv Net  Model architecture definition<jupyter_code>model = Sequential()
model.add(Convolution2D(32, 3, 3, input_shape=(img_width, img_height,3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Training<jupyter_code>nb_epoch = 30
nb_train_samples = 2048
nb_validation_samples = 832
model.fit_generator(
        train_generator,
        samples_per_epoch=nb_train_samples,
        nb_epoch=nb_epoch,
        validation_data=validation_generator,
        nb_val_samples=nb_validation_samples)
model.save_weights('models/basic_cnn_20_epochs.h5')
#model.load_weights('models_trained/basic_cnn_20_epochs.h5')<jupyter_output><empty_output><jupyter_text>If your model successfully runs at one epoch, go back and it for 30 epochs by changing nb_epoch above.  I was able to get to an val_acc of 0.71 at 30 epochs.A copy of a pretrained network is available in the pretrained folder.  Evaluating on validation set Computing loss and accuracy :<jupyter_code>model.evaluate_generator(validation_generator, nb_validation_samples)<jupyter_output><empty_output><jupyter_text>Evolution of accuracy on training (blue) and validation (green) sets for 1 to 32 epochs : ![Accuracy evolution](pictures/scores_no_dataaugmentation.png) **After ~10 epochs the neural network reach ~70% accuracy. We can witness overfitting, no progress is made over validation set in the next epochs**  Data augmentation for improving the model By applying random transformation to our train set, we artificially enhance our dataset with new unseen images.  This will hopefully reduce overfitting and allows better generalization capability for our network. Example of data augmentation applied to a picture:![Example of data augmentation applied to a picture](pictures/cat_data_augmentation.png)<jupyter_code>train_datagen_augmented = ImageDataGenerator(
        rescale=1./255,        # normalize pixel values to [0,1]
        shear_range=0.2,       # randomly applies shearing transformation
        zoom_range=0.2,        # randomly applies shearing transformation
        horizontal_flip=True)  # randomly flip the images

# same code as before
train_generator_augmented = train_datagen_augmented.flow_from_directory(
        train_data_dir,
        target_size=(img_width, img_height),
        batch_size=32,
        class_mode='binary')
model.fit_generator(
        train_generator_augmented,
        samples_per_epoch=nb_train_samples,
        nb_epoch=nb_epoch,
        validation_data=validation_generator,
        nb_val_samples=nb_validation_samples)
model.save_weights('models/augmented_30_epochs.h5')
#model.load_weights('models_trained/augmented_30_epochs.h5')<jupyter_output><empty_output><jupyter_text>Evaluating on validation set Computing loss and accuracy :<jupyter_code>model.evaluate_generator(validation_generator, nb_validation_samples)<jupyter_output><empty_output><jupyter_text>Evolution of accuracy on training (blue) and validation (green) sets for 1 to 100 epochs : ![Accuracy evolution](pictures/scores_with_dataaugmentation.png) **Thanks to data-augmentation, the accuracy on the validation set improved to ~80%**  Using a pre-trained model The process of training a convolutionnal neural network can be very time-consuming and require a lot of datas.  We can go beyond the previous models in terms of performance and efficiency by using a general-purpose, pre-trained image classifier.  This example uses VGG16, a model trained on the ImageNet dataset - which contains millions of images classified in 1000 categories. On top of it, we add a small multi-layer perceptron and we train it on our dataset.  VGG16 + small MLP![VGG16 + Dense layers Schema](pictures/vgg16_original.png)  VGG16 model architecture definition<jupyter_code>model_vgg = Sequential()
model_vgg.add(ZeroPadding2D((1, 1), input_shape=(img_width, img_height,3)))
model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))<jupyter_output><empty_output><jupyter_text>Loading VGG16 weightsThis part is a bit complicated because the structure of our model is not exactly the same as the one used when training weights.  Otherwise, we would use the `model.load_weights()` method. *Note : the VGG16 weights file (~500MB) is not included in this repository. You can download from here :  https://gist.github.com/baraldilorenzo/07d7802847aaad0a35d3*<jupyter_code>import h5py
f = h5py.File('models/vgg/vgg16_weights.h5')
for k in range(f.attrs['nb_layers']):
    if k >= len(model_vgg.layers) - 1:
        # we don't look at the last two layers in the savefile (fully-connected and activation)
        break
    g = f['layer_{}'.format(k)]
    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
    layer = model_vgg.layers[k]

    if layer.__class__.__name__ in ['Convolution1D', 'Convolution2D', 'Convolution3D', 'AtrousConvolution2D']:
        weights[0] = np.transpose(weights[0], (2, 3, 1, 0))

    layer.set_weights(weights)

f.close()<jupyter_output><empty_output><jupyter_text>Using the VGG16 model to process samples<jupyter_code>train_generator_bottleneck = datagen.flow_from_directory(
        train_data_dir,
        target_size=(img_width, img_height),
        batch_size=32,
        class_mode=None,
        shuffle=False)

validation_generator_bottleneck = datagen.flow_from_directory(
        validation_data_dir,
        target_size=(img_width, img_height),
        batch_size=32,
        class_mode=None,
        shuffle=False)<jupyter_output>Found 2048 images belonging to 2 classes.
Found 832 images belonging to 2 classes.<jupyter_text>This is a long process, so we save the output of the VGG16 once and for all.<jupyter_code>bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck, nb_train_samples)
np.save(open('models/bottleneck_features_train.npy', 'wb'), bottleneck_features_train)
bottleneck_features_validation = model_vgg.predict_generator(validation_generator_bottleneck, nb_validation_samples)
np.save(open('models/bottleneck_features_validation.npy', 'wb'), bottleneck_features_validation)<jupyter_output><empty_output><jupyter_text>Now we can load it...<jupyter_code>train_data = np.load(open('models/bottleneck_features_train.npy', 'rb'))
train_labels = np.array([0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2))

validation_data = np.load(open('models/bottleneck_features_validation.npy', 'rb'))
validation_labels = np.array([0] * (nb_validation_samples // 2) + [1] * (nb_validation_samples // 2))<jupyter_output><empty_output><jupyter_text>And define and train the custom fully connected neural network :<jupyter_code>model_top = Sequential()
model_top.add(Flatten(input_shape=train_data.shape[1:]))
model_top.add(Dense(256, activation='relu'))
model_top.add(Dropout(0.5))
model_top.add(Dense(1, activation='sigmoid'))

model_top.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])
nb_epoch=40
model_top.fit(train_data, train_labels,
          nb_epoch=nb_epoch, batch_size=32,
          validation_data=(validation_data, validation_labels))<jupyter_output>Train on 2048 samples, validate on 832 samples
Epoch 1/40
2048/2048 [==============================] - 0s - loss: 0.9784 - acc: 0.6855 - val_loss: 0.4140 - val_acc: 0.8077
Epoch 2/40
2048/2048 [==============================] - 0s - loss: 0.4831 - acc: 0.7891 - val_loss: 0.3646 - val_acc: 0.8365
Epoch 3/40
2048/2048 [==============================] - 0s - loss: 0.3974 - acc: 0.8271 - val_loss: 0.3369 - val_acc: 0.8558
Epoch 4/40
2048/2048 [==============================] - 0s - loss: 0.3438 - acc: 0.8525 - val_loss: 0.3136 - val_acc: 0.8594
Epoch 5/40
2048/2048 [==============================] - 0s - loss: 0.3131 - acc: 0.8647 - val_loss: 0.5777 - val_acc: 0.7873
Epoch 6/40
2048/2048 [==============================] - 0s - loss: 0.2883 - acc: 0.8823 - val_loss: 0.2913 - val_acc: 0.8786
Epoch 7/40
2048/2048 [==============================] - 0s - loss: 0.2321 - acc: 0.9048 - val_loss: 0.3724 - val_acc: 0.8462
Epoch 8/40
2048/2048 [==============================] - 0s - loss: 0.2057 - ac[...]<jupyter_text>The training process of this small neural network is very fast : ~2s per epoch<jupyter_code>model_top.save_weights('models/bottleneck_40_epochs.h5')<jupyter_output><empty_output><jupyter_text>Bottleneck model evaluation<jupyter_code>#model_top.load_weights('models/with-bottleneck/1000-samples--100-epochs.h5')
#model_top.load_weights('/notebook/Data1/Code/keras-workshop/models/with-bottleneck/1000-samples--100-epochs.h5')<jupyter_output><empty_output><jupyter_text>Loss and accuracy :<jupyter_code>model_top.evaluate(validation_data, validation_labels)<jupyter_output>704/832 [========================>.....] - ETA: 0s<jupyter_text>Evolution of accuracy on training (blue) and validation (green) sets for 1 to 32 epochs : ![Accuracy evolution](pictures/scores_with_bottleneck.png) **We reached a 90% accuracy on the validation after ~1m of training (~20 epochs) and 8% of the samples originally available on the Kaggle competition !**<jupyter_code>##Fine-tuning the top layers of a a pre-trained network<jupyter_output><empty_output><jupyter_text>Start by instantiating the VGG base and loading its weights.<jupyter_code>model_vgg = Sequential()
model_vgg.add(ZeroPadding2D((1, 1), input_shape=(img_width, img_height,3)))
model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(64, 3, 3, activation='relu', name='conv1_2'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(128, 3, 3, activation='relu', name='conv2_2'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_2'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(256, 3, 3, activation='relu', name='conv3_3'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_2'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv4_3'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))

model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_1'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_2'))
model_vgg.add(ZeroPadding2D((1, 1)))
model_vgg.add(Convolution2D(512, 3, 3, activation='relu', name='conv5_3'))
model_vgg.add(MaxPooling2D((2, 2), strides=(2, 2)))
import h5py
f = h5py.File('models/vgg/vgg16_weights.h5')
for k in range(f.attrs['nb_layers']):
    if k >= len(model_vgg.layers) - 1:
        # we don't look at the last two layers in the savefile (fully-connected and activation)
        break
    g = f['layer_{}'.format(k)]
    weights = [g['param_{}'.format(p)] for p in range(g.attrs['nb_params'])]
    layer = model_vgg.layers[k]

    if layer.__class__.__name__ in ['Convolution1D', 'Convolution2D', 'Convolution3D', 'AtrousConvolution2D']:
        weights[0] = np.transpose(weights[0], (2, 3, 1, 0))

    layer.set_weights(weights)

f.close()<jupyter_output><empty_output><jupyter_text>Build a classifier model to put on top of the convolutional model. For the fine tuning, we start with a fully trained-classifer. We will use the weights from the earlier model. And then we will add this model on top of the convolutional base.<jupyter_code>top_model = Sequential()
top_model.add(Flatten(input_shape=model_vgg.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))

top_model.load_weights('models/bottleneck_40_epochs.h5')

model_vgg.add(top_model)<jupyter_output><empty_output><jupyter_text>For fine turning, we only want to train a few layers.  This line will set the first 25 layers (up to the conv block) to non-trainable.<jupyter_code>for layer in model_vgg.layers[:25]:
    layer.trainable = False
# compile the model with a SGD/momentum optimizer
# and a very slow learning rate.
model_vgg.compile(loss='binary_crossentropy',
              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])
# prepare data augmentation configuration  . . . do we need this?
train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_data_dir,
        target_size=(img_height, img_width),
        batch_size=32,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_data_dir,
        target_size=(img_height, img_width),
        batch_size=32,
        class_mode='binary')
# fine-tune the model
model_vgg.fit_generator(
        train_generator,
        samples_per_epoch=nb_train_samples,
        nb_epoch=nb_epoch,
        validation_data=validation_generator,
        nb_val_samples=nb_validation_samples)
model_vgg.save_weights('models/finetuning_20epochs_vgg.h5')
model_vgg.load_weights('models/finetuning_20epochs_vgg.h5')<jupyter_output><empty_output><jupyter_text>Evaluating on validation set Computing loss and accuracy :<jupyter_code>model_vgg.evaluate_generator(validation_generator, nb_validation_samples)
model.evaluate_generator(validation_generator, nb_validation_samples)
model_top.evaluate(validation_data, validation_labels)<jupyter_output>704/832 [========================>.....] - ETA: 0s
{'fraction_non_alphanumeric': 0.13056978730010868, 'fraction_numerical': 0.04864669047249392, 'mean_word_length': 5.430615640599002, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 86, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '664774', 'license': 'MIT', 'n_tokens_mistral': 7077, 'n_tokens_neox': 6458, 'n_words': 1525, 'path': '2-convolutional-neural-networks/how-to-make-an-image-classifier/demo.ipynb', 'repo_name': 'vanyaland/deep-learning-foundation', 'size': '19345'}