<jupyter_start><jupyter_text>EXPERIMENT 6Applying Fine Tunning on DenseNet201 network to the CIFAR 10 dataset.  FINE TUNNINGThe task of fine-tuning a network is to tweak the parameters of an already trained network so that it adapts to the new task at hand.The initial layers learn very general features and as we go higher up the network, the layers tend to learn patterns more specific to the task it is being trained on. Thus, for fine-tuning, we keep the initial layers intact and retrain the later layers of the model.  DatasetCIFAR-10 dataset The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class.http://www.cs.utoronto.ca/~kriz/cifar.html  Importing Libraries & Loading the Dataset<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from keras.datasets import cifar10
import keras

#Load the dataset:
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
print("There are {} train images and {} test images.".format(X_train.shape[0], X_test.shape[0]))
print('There are {} unique classes to predict.'.format(np.unique(y_train).shape[0]))<jupyter_output>There are 50000 train images and 10000 test images.
There are 10 unique classes to predict.<jupyter_text>One-hot encoding the labels<jupyter_code>num_classes = 10
from keras.utils import np_utils
y_train = np_utils.to_categorical(y_train, num_classes)
y_test = np_utils.to_categorical(y_test, num_classes)<jupyter_output><empty_output><jupyter_text>Visualizing & Displaying the first eight images in the training data<jupyter_code>fig = plt.figure(figsize=(10, 10))

for i in range(1, 9):
    img = X_train[i-1]
    fig.add_subplot(2, 4, i)
    plt.imshow(img)<jupyter_output><empty_output><jupyter_text>Building up a Sequential model<jupyter_code>#Importing the necessary libraries 
from keras.models import Sequential
from keras.layers import Dense, Conv2D, MaxPooling2D
from keras.layers import Dropout, Flatten, GlobalAveragePooling2D


model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu',input_shape = X_train.shape[1:]))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(GlobalAveragePooling2D())
model.add(Dense(10, activation='softmax'))
model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_13 (Conv2D)           (None, 30, 30, 32)        896       
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 (None, 15, 15, 32)        0         
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 13, 13, 32)        9248      
_________________________________________________________________
max_pooling2d_9 (MaxPooling2 (None, 6, 6, 32)          0         
_________________________________________________________________
conv2d_15 (Conv2D)           (None, 4, 4, 64)          18496     
_________________________________________________________________
max_pooling2d_10 (MaxPooling (None, 2, 2, 64)          0         
_________________________________________________________________
global_ave[...]<jupyter_text>Compliing the Model<jupyter_code>model.compile(loss='binary_crossentropy', optimizer='adam',
              metrics=['accuracy'])
X_train_scratch = X_train/255.
X_test_scratch = X_test/255.<jupyter_output><empty_output><jupyter_text>Creating a checkpointer A checkpointer is used to save the weights of the best model (i.e. the model with minimum loss).<jupyter_code>checkpointer =keras.callbacks.ModelCheckpoint(filepath='scratchmodel.best.hdf5', 
                               verbose=1,save_best_only=True)
# keras.callbacks.ModelCheckpoint(filepath ='scratchmodel.best.hdf5', monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)
#Fitting the model on the train data and labels.
model.fit(X_train, y_train, batch_size=32, epochs=10, 
          verbose=1, callbacks=[checkpointer], validation_split=0.2, shuffle=True)<jupyter_output>Train on 40000 samples, validate on 10000 samples
Epoch 1/10
40000/40000 [==============================] - 17s 416us/step - loss: 0.4729 - acc: 0.8898 - val_loss: 0.2475 - val_acc: 0.9060

Epoch 00001: val_loss improved from inf to 0.24749, saving model to scratchmodel.best.hdf5
Epoch 2/10
40000/40000 [==============================] - 12s 293us/step - loss: 0.2258 - acc: 0.9131 - val_loss: 0.2192 - val_acc: 0.9146

Epoch 00002: val_loss improved from 0.24749 to 0.21916, saving model to scratchmodel.best.hdf5
Epoch 3/10
40000/40000 [==============================] - 12s 293us/step - loss: 0.2070 - acc: 0.9199 - val_loss: 0.1976 - val_acc: 0.9233

Epoch 00003: val_loss improved from 0.21916 to 0.19756, saving model to scratchmodel.best.hdf5
Epoch 4/10
40000/40000 [==============================] - 12s 299us/step - loss: 0.1922 - acc: 0.9253 - val_loss: 0.1915 - val_acc: 0.9256

Epoch 00004: val_loss improved from 0.19756 to 0.19146, saving model to scratchmodel.best.hdf5
Epoch 5/10
400[...]<jupyter_text>Evaluating the model on the test data & Printing the accuracy<jupyter_code>score = model.evaluate(X_test, y_test)

#Accuracy on test data
print('Accuracy on the Test Images: ', score[1])
# from keras_applications import densenet
# from keras.applications.imagenet_utils import preprocess_input as _preprocess_input
# from keras.applications import DenseNet201
# from skimage import data, io, filters, transform


#Importing the Densenet201 model
from keras.applications.densenet import DenseNet201, preprocess_input


#Loading the Densenet201 model with pre-trained ImageNet weights
model = DenseNet201(weights='imagenet', include_top=False, input_shape=(32, 32, 3))
def DenseNetImageNet201(input_shape=None,
                        bottleneck=True,
                        reduction=0.5,
                        dropout_rate=0.0,
                        weight_decay=1e-4,
                        include_top=True,
                        weights=None,
                        input_tensor=None,
                        pooling=None,
                        classes=1000,
                        activation='softmax'):
    return DenseNet(input_shape, depth=201, nb_dense_block=4, growth_rate=32,
                    nb_filter=64, nb_layers_per_block=[6, 12, 48, 32],
                    bottleneck=bottleneck, reduction=reduction,
                    dropout_rate=dropout_rate, weight_decay=weight_decay,
                    subsample_initial_block=True, include_top=include_top,
                    weights=weights, input_tensor=input_tensor,
                    pooling=pooling, classes=classes, activation=activation)<jupyter_output><empty_output><jupyter_text>Reshaping & Preprocessing the training data<jupyter_code># from keras.applications.imagenet_utils import preprocess_input as _preprocess_input
import numpy as np
import skimage 
from scipy.misc import imresize

#Preprocessing the data, so that it can be fed to the pre-trained ResNet50 model. 
densenet_train_input = preprocess_input(X_train)

#Creating bottleneck features for the training data
train_features = model.predict(densenet_train_input)

#Saving the bottleneck features
np.savez('densenet_train_input', features=train_features)<jupyter_output><empty_output><jupyter_text>Reshaping & Preprocessing the testing data<jupyter_code>#Preprocessing the data, so that it can be fed to the pre-trained ResNet50 model.
densenet_test_input = preprocess_input(X_test)

#Creating bottleneck features for the testing data
test_features = model.predict(densenet_test_input)

#Saving the bottleneck features
np.savez('densenet_features_test', features=test_features)
#print(X_train.shape[1:])
print(X_train.shape[1:])
from keras.callbacks import LearningRateScheduler
from keras import regularizers
from keras.layers import Dense, Activation, Flatten, Dropout, BatchNormalization
def lr_schedule(epoch):
    lrate = 0.001
    if epoch > 75:
        lrate = 0.0005
    if epoch > 100:
        lrate = 0.0003
    return lrate<jupyter_output><empty_output><jupyter_text>Fine tunning the model<jupyter_code>weight_decay = 1e-4
model = Sequential()
model.add(Conv2D(32, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay), input_shape=X_train.shape[1:]))
model.add(Activation('elu'))
model.add(BatchNormalization())
model.add(Conv2D(32, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))
model.add(Activation('elu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.2))
 
model.add(Conv2D(64, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))
model.add(Activation('elu'))
model.add(BatchNormalization())
model.add(Conv2D(64, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))
model.add(Activation('elu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.3))
 
model.add(Conv2D(128, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))
model.add(Activation('elu'))
model.add(BatchNormalization())
model.add(Conv2D(128, (3,3), padding='same', kernel_regularizer=regularizers.l2(weight_decay)))
model.add(Activation('elu'))
model.add(BatchNormalization())
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.4))
 
model.add(Flatten())
model.add(Dense(num_classes, activation='softmax'))
 
model.summary()<jupyter_output>_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_7 (Conv2D)            (None, 32, 32, 32)        896       
_________________________________________________________________
activation_1 (Activation)    (None, 32, 32, 32)        0         
_________________________________________________________________
batch_normalization_1 (Batch (None, 32, 32, 32)        128       
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 32, 32, 32)        9248      
_________________________________________________________________
activation_2 (Activation)    (None, 32, 32, 32)        0         
_________________________________________________________________
batch_normalization_2 (Batch (None, 32, 32, 32)        128       
_________________________________________________________________
max_poolin[...]<jupyter_text>Compliing the Model with DenseNet201<jupyter_code>model.compile(loss='categorical_crossentropy', optimizer='adam', 
              metrics=['accuracy'])
#Creating a checkpointer 
checkpointer =keras.callbacks.ModelCheckpoint(filepath='scratchmodel.best.hdf5', 
                               verbose=1,save_best_only=True)
#Fitting the model on the train data and labels.
model.fit(X_train, y_train, batch_size=32, epochs=10, 
          verbose=1, callbacks=[checkpointer], validation_split=0.2, shuffle=True)<jupyter_output>Train on 40000 samples, validate on 10000 samples
Epoch 1/10
40000/40000 [==============================] - 50s 1ms/step - loss: 1.7532 - acc: 0.4640 - val_loss: 1.2133 - val_acc: 0.5905

Epoch 00001: val_loss improved from inf to 1.21334, saving model to scratchmodel.best.hdf5
Epoch 2/10
40000/40000 [==============================] - 46s 1ms/step - loss: 1.1185 - acc: 0.6364 - val_loss: 0.9951 - val_acc: 0.6829

Epoch 00002: val_loss improved from 1.21334 to 0.99510, saving model to scratchmodel.best.hdf5
Epoch 3/10
40000/40000 [==============================] - 46s 1ms/step - loss: 0.9460 - acc: 0.6967 - val_loss: 0.9960 - val_acc: 0.6833

Epoch 00003: val_loss did not improve from 0.99510
Epoch 4/10
40000/40000 [==============================] - 46s 1ms/step - loss: 0.8454 - acc: 0.7349 - val_loss: 0.9138 - val_acc: 0.7154

Epoch 00004: val_loss improved from 0.99510 to 0.91378, saving model to scratchmodel.best.hdf5
Epoch 5/10
40000/40000 [==============================] - 46s 1ms/[...]<jupyter_text>Evaluating the model on the test data & Printing the accuracy using the DenseNet201<jupyter_code>scores = model.evaluate(X_test, y_test, verbose=1)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])<jupyter_output>10000/10000 [==============================] - 3s 283us/step
Test loss: 0.76271309633255
Test accuracy: 0.805
{'fraction_non_alphanumeric': 0.12679938744257274, 'fraction_numerical': 0.06875957120980092, 'mean_word_length': 4.7310223782360685, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '60283', 'license': 'MIT', 'n_tokens_mistral': 4629, 'n_tokens_neox': 3957, 'n_words': 1011, 'path': 'Imagenet Image Classification using Keras/Fine_Tunning.ipynb', 'repo_name': 'RajeshreeKale/PROJECTS', 'size': '13060'}