<jupyter_start><jupyter_text>Feature ExtractionWe will extract features from pretrained models like VGG-16, VGG-19, ResNet-50, InceptionV3 and MobileNet and benchmark them using the udset dataset. Dataset:UC Merced Land Use DatasetThis is a 21 class land use image dataset meant for research purposes.There are 100 images for each class.Each image measures 256x256 pixels. (*We resize the images to 224x224 to be compatible with the input dimensions of the various models used in this project)The images were manually extracted from large images from the USGS National Map Urban Area Imagery collection for various urban areas around the country. The pixel resolution of this public domain imagery is 1 foot.Credits for Dataset Curation:<NAME> and <NAME>, "Bag-Of-Visual-Words and Spatial Extensions for Land-Use Classification," ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (ACM GIS), 2010.<jupyter_code>#from google.colab import drive
#drive.mount('/content/drive')
!pip3 install sklearn
!mkdir -p ../../datasets
!pip install gdown
!gdown http://weegee.vision.ucmerced.edu/datasets/UCMerced_LandUse.zip --output ../../datasets/UCMerced_LandUse.zip
!unzip ../../datasets/UCMerced_LandUse.zip -d ../../datasets/
!mv ../../datasets/UCMerced_LandUse/Images ../../datasets/udset
import numpy as np
from numpy.linalg import norm
import pickle
from tqdm import tqdm, tqdm_notebook
import os
import random
import time
import math
import tensorflow
from tensorflow.keras.preprocessing import image
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.applications.vgg19 import VGG19
from tensorflow.keras.applications.mobilenet import MobileNet
from tensorflow.keras.applications.inception_v3 import InceptionV3
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Flatten, Dense, Dropout, GlobalAveragePooling2D
import numpy as np
import pickle
from tqdm import tqdm, tqdm_notebook
import random
import time
from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
import PIL
from PIL import Image
from sklearn.neighbors import NearestNeighbors

import glob
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline<jupyter_output><empty_output><jupyter_text>We define a helper function that allows us to choose any pretrained model with all the necessary details for our experiments.<jupyter_code>def model_picker(name):
    if (name == 'vgg16'):
        model = VGG16(weights='imagenet',
                      include_top=False,
                      input_shape=(224, 224, 3),
                      pooling='max')
    elif (name == 'vgg19'):
        model = VGG19(weights='imagenet',
                      include_top=False,
                      input_shape=(224, 224, 3),
                      pooling='max')
    elif (name == 'mobilenet'):
        model = MobileNet(weights='imagenet',
                          include_top=False,
                          input_shape=(224, 224, 3),
                          pooling='max',
                          depth_multiplier=1,
                          alpha=1)
    elif (name == 'inception'):
        model = InceptionV3(weights='imagenet',
                            include_top=False,
                            input_shape=(224, 224, 3),
                            pooling='max')
    elif (name == 'resnet'):
        model = ResNet50(weights='imagenet',
                         include_top=False,
                         input_shape=(224, 224, 3),
                         pooling='max')
    elif (name == 'xception'):
        model = Xception(weights='imagenet',
                         include_top=False,
                         input_shape=(224, 224, 3),
                         pooling='max')
    else:
        print("Specified model not available")
    return model<jupyter_output><empty_output><jupyter_text>Now, let's put our function to use.<jupyter_code>model_architecture = 'resnet'
model = model_picker(model_architecture)<jupyter_output>Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5
94773248/94765736 [==============================] - 1s 0us/step
94781440/94765736 [==============================] - 1s 0us/step<jupyter_text>Let's define a function to extract image features given an image and a model.<jupyter_code>def extract_features(img_path, model):
    input_shape = (224, 224, 3)
    img = image.load_img(img_path, target_size=(input_shape[0], input_shape[1]))
    img_array = image.img_to_array(img)
    expanded_img_array = np.expand_dims(img_array, axis=0)
    preprocessed_img = preprocess_input(expanded_img_array)
    features = model.predict(preprocessed_img)
    flattened_features = features.flatten()
    normalized_features = flattened_features / norm(flattened_features)
    return normalized_features<jupyter_output><empty_output><jupyter_text>The time taken to extract features is dependent on a few factors such as image size, computing power etc. A better benchmark would be running the network over an entire dataset. A simple change to the existing code will allow this.Let's make a handy function to recursively get all the image files under a root directory.<jupyter_code>extensions = ['.jpg', '.JPG', '.jpeg', '.JPEG', '.png', '.PNG', '.tif', '.TIF']

def get_file_list(root_dir):
    file_list = []
    for root, directories, filenames in os.walk(root_dir):
        for filename in filenames:
            if any(ext in filename for ext in extensions):
                file_list.append(os.path.join(root, filename))
    return file_list<jupyter_output><empty_output><jupyter_text>Now, let's run the extraction over the entire dataset and time it.<jupyter_code># path to the your datasets
root_dir = '../../datasets/udset'
filenames = sorted(get_file_list(root_dir))

#feature_list = []
#for i in tqdm_notebook(range(len(filenames))):
#    feature_list.append(extract_features(filenames[i], model))<jupyter_output><empty_output><jupyter_text>Now let's try the same with the Keras Image Generator functions.<jupyter_code>batch_size = 64
datagen = tensorflow.keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input)

generator = datagen.flow_from_directory(root_dir,
                                        target_size=(224, 224),
                                        batch_size=batch_size,
                                        class_mode=None,
                                        shuffle=False)

num_images = len(generator.filenames)
num_epochs = int(math.ceil(num_images / batch_size))

start_time = time.time()
feature_list = []
feature_list = model.predict_generator(generator, num_epochs)
end_time = time.time()
for i, features in enumerate(feature_list):
    feature_list[i] = features / norm(features)

feature_list = feature_list.reshape(num_images, -1)

print("Num images   = ", len(generator.classes))
print("Shape of feature_list = ", feature_list.shape)
print("Time taken in sec = ", end_time - start_time)<jupyter_output>Num images   =  2100
Shape of feature_list =  (2100, 2048)
Time taken in sec =  48.58677101135254<jupyter_text>Some benchmarks on different model architectures to see relative speedsKeeping batch size of 64, benchmarking the different models|Model |items/second ||-|-|-|| VGG19     | 31.06 || VGG16     | 28.16 | | Resnet50  | 28.48 | | Inception | 20.07 || Mobilenet | 13.45 | Let's save the features as intermediate files to use later.<jupyter_code>filenames = [root_dir + '/' + s for s in generator.filenames]
!mkdir data
pickle.dump(generator.classes, open('./data/class_ids-udset.pickle','wb'))
pickle.dump(filenames, open('./data/filenames-udset.pickle', 'wb'))
pickle.dump(feature_list, open('./data/features-udset-' + model_architecture + '.pickle', 'wb'))<jupyter_output><empty_output><jupyter_text>Let's train a finetuned model as well and save the features for that as well.<jupyter_code>TRAIN_SAMPLES = 2100
NUM_CLASSES = 21
IMG_WIDTH, IMG_HEIGHT = 224, 224
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input,
                                   rotation_range=20,
                                   width_shift_range=0.2,
                                   height_shift_range=0.2,
                                   zoom_range=0.2)
train_generator = train_datagen.flow_from_directory(root_dir,
                                                    target_size=(IMG_WIDTH,
                                                                 IMG_HEIGHT),
                                                    batch_size=batch_size,
                                                    shuffle=True,
                                                    seed=12345,
                                                    class_mode='categorical')
def model_maker():
    base_model = ResNet50(include_top=False,
                           input_shape=(IMG_WIDTH, IMG_HEIGHT, 3))
    for layer in base_model.layers[:]:
        layer.trainable = False
    input = Input(shape=(IMG_WIDTH, IMG_HEIGHT, 3))
    custom_model = base_model(input)
    custom_model = GlobalAveragePooling2D()(custom_model)
    custom_model = Dense(64, activation='relu')(custom_model)
    custom_model = Dropout(0.5)(custom_model)
    predictions = Dense(NUM_CLASSES, activation='softmax')(custom_model)
    return Model(inputs=input, outputs=predictions)
model_finetuned = model_maker()
model_finetuned.compile(loss='categorical_crossentropy',
              optimizer=tensorflow.keras.optimizers.Adam(0.001),
              metrics=['acc'])
model_finetuned.fit_generator(
    train_generator,
    steps_per_epoch=math.ceil(float(TRAIN_SAMPLES) / batch_size),
    epochs=10)
model_finetuned.save('./data/model-finetuned.h5')
start_time = time.time()
feature_list_finetuned = []
feature_list_finetuned = model_finetuned.predict_generator(generator, num_epochs)
end_time = time.time()

for i, features_finetuned in enumerate(feature_list_finetuned):
    feature_list_finetuned[i] = features_finetuned / norm(features_finetuned)

feature_list = feature_list_finetuned.reshape(num_images, -1)

print("Num images   = ", len(generator.classes))
print("Shape of feature_list = ", feature_list.shape)
print("Time taken in sec = ", end_time - start_time)
pickle.dump(feature_list, open('./data/features-udset-resnet-finetuned.pickle', 'wb'))
print(feature_list)
print(type(feature_list))

#!cp -a data/* drive/MyDrive/c99t1/ save to gdirve (should be mounted first for later use)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09835455982151158, 'fraction_numerical': 0.024077344984661152, 'mean_word_length': 3.163312693498452, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 21, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '249261', 'license': 'MIT', 'n_tokens_mistral': 3363, 'n_tokens_neox': 3105, 'n_words': 782, 'path': '1_feature_extract.ipynb', 'repo_name': 'shankch/reverse-image', 'size': '10764'}