<jupyter_start><jupyter_text>DocuSortTo begin, we need to import the libraries we need for our task. We'll use Keras to build and train our model and matplotlib to plot our model performance, and we'll use numpy for the rest.<jupyter_code>from tensorflow.keras.applications.xception import Xception
from tensorflow.keras.preprocessing import image
from tensorflow.keras.applications.xception import preprocess_input, decode_predictions
from keras.preprocessing import image
from keras.applications.imagenet_utils import preprocess_input
import pandas 
import tensorflow.keras as keras
from tensorflow.keras import backend as K
import numpy as np
from IPython.display import display
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d 
%matplotlib inline<jupyter_output><empty_output><jupyter_text>The Dataset    Our dataset for this problem is a subset of the RVL-CDIP (Ryerson Vision Lab Complex Document Information Processing) dataset. The original dataset consisted of 400,000 grayscale images in 16 classes (shown below), with 25,000 images per class, and for our subset, we used only 1,250 of these images, which were chosen at random. Then, we created a file that contained all of the corresponding labels. We chose to use a small subset of this massive dataset because of storage constraints, and it made reading in the data much simpler since the images in the original dataset were scattered between multiple folders. Because we chose the images to use at random, there was added bias due to some classes being underrepresented in the training data but being overrepresented in the testing data.![Screen Shot 2021-04-20 at 1.59.40 PM.png](attachment:1b401edf-fdc9-4602-a50e-05c97dcf5be8.png) Reading in the DataIn order to read in the data, you need to put the folder containing the images (Project_images)and the file containing the data labels (Project_labels) in the same folder as the demo iPython notebook. This is important because, in order to read in the image data, we use the path to the individual images to grab each one. Once every image has been loaded, we're left with a 4D tensor that we will be using as input for our networks. The first dimension represents the number of images that we read in (1250), while our second and third dimensions represent the size of the image being loaded in. In this case, the size of the images is 224×224 pixels, and although the images in our dataset are black and white, the network we're using for transfer learning (VGG16) requires that the input images are read in as RBG. Therefore, our fourth dimension is 3 to read in the three color channels (red, blue, and green).<jupyter_code>def grab_image(img_path):
    img = image.load_img(img_path, target_size=(224, 224)) 
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x = preprocess_input(x)
    return x

# Grab training images
image_data = np.concatenate( 
    [grab_image('Project_images/image%d.tif'%(i))
    for i in range(1,1251)]) 
image_data.shape
# Split image data into training and testing sets (84%, 16%)
x_train = image_data[0:1050]
x_test = image_data[1050:1250]

print("x_train:",x_train.shape)
print("x_test",x_test.shape)<jupyter_output>x_train: (1050, 224, 224, 3)
x_test (200, 224, 224, 3)<jupyter_text>For the data labels, we load the .csv file into our program as an array using numpy's loadtxt function. Then, we convert the class vector (contains classes 0-15) to categorical assignments.<jupyter_code># Read in data labels
label_data = np.loadtxt('Project_labels.csv')

# One-hot encode class vector [0-15] 
label_data = keras.utils.to_categorical(label_data)
display(label_data.shape)
display(label_data[0:10])
# Split label data into training and testing sets (84% 16%)
y_train = label_data[0:1050]
y_test = label_data[1050:1250]
print("y_train:",y_train.shape)
print("y_test",y_test.shape)
## Standardize the -input- data between [-1.0,1.0] (real)
## instead of the default 0-255 (integer)
x_train = x_train.astype('float32')/255.
x_test = x_test.astype('float32')/255.

display(x_train.shape)
display(y_train.shape)
display(x_test.shape)
display(y_test.shape)<jupyter_output><empty_output><jupyter_text>Simple CNNWe decided to use our smaller dataset as an opportunity to test the efficacy of applying transfer learning to a convolution neural network (CNN). Our network architecture was inspired by the work of Ryerson University. First, we created a simple CNN in order to test its performance on our small dataset. Then, applied transfer learning to a CNN, and we gathered data about its performance in order to compare it with that of the simple CNN. We anticipated that the transfer learning would yield greater training and testing accuracy than the simple CNN by itself. However, we didn't expect either of our two models to perform better than those used in the Ryerson University study, which were trained on the full RVL-CDIP dataset. The simple CNN model used in the Ryerson University study included two convolutional layers and threefully-connected layers, with  pooling, ReLU, and drop-out, which was similar to the simple network that we built. However, their network took in images of size 150×150 as input, while our network took them in as size 224×224.  CNNs for Image ClassificationWhen you see a picture of a cat, how do you know it's a cat? In the image, you might recognize there's a cat because you notice its key features: tall, pointy ears, a long tail, etc. Identifying key features of an image is what allows you to classify it, and CNNs achieve this in a similar manner. Images are read into the computer as pixels, and in order to preserve the spatial relationship of those pixels, CNNs read in image data as 2D tensors rather than trying to flatten the representation of the image into a one dimensional tensor. Then, they use kernels to scan patches of the image in order to identify where specific features might be present. After that, the network uses pooling to actually detect those feautures regardless of where they appear in the image, which eliminates translational variance.  Training the ModelThis model is _large_ (over 8 million parameters!). Therefore, training takes about 30 minutes because each epoch takes about 3 minutes. Similarly, the second model that uses transfer learning is large and will require approximately 10 minutes per epoch.<jupyter_code># Build model
model=keras.Sequential()

model.add(keras.layers.Conv2D(36,kernel_size=(6,6),
                              activation='relu',input_shape=[x_train.shape[1],
                                                             x_train.shape[2],
                                                             x_train.shape[3]]))
model.add(keras.layers.Conv2D(40,(6,6),activation='relu'))
model.add(keras.layers.MaxPooling2D(pool_size=(2,2)))
model.add(keras.layers.Dropout(0.25))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(40,activation='relu'))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(y_train.shape[1],activation='softmax'))

model.compile(loss = keras.losses.CategoricalCrossentropy(),
              optimizer = keras.optimizers.Adam(),
              metrics=[keras.metrics.CategoricalAccuracy()])
model.summary()
#Train model
batch_size = 20
epochs = 10
history = model.fit(x_train, y_train,
                    batch_size = batch_size,
                    epochs = epochs,
                    verbose = 1,
                    validation_split = 0.2)
# Testing
results = model.evaluate(x_test, y_test, verbose=1)
print('Test loss:', results[0])
print('Test accuracy:', results[1])
plt.figure()

# Summarize history for accuracy  
plt.subplot(211)
plt.plot(history.history['categorical_accuracy'])
plt.plot(history.history['val_categorical_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['training','validation'],loc='upper left')

# Summarize history for loss  
plt.subplot(212)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['training','validation'],loc='upper left')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>After training, we can see that the model experienced _extreme_ levels of overfitting, meaning the network memorized the training data and was unable to generalize when it was presented with images it had never seen before. Therefore, the network performed poorly on the validation and testing datasets.   CNN with Transfer LearningAlthough CNNs are ideal for image classification, they can only perform as well as the weights they're initialized with. The simple CNN we built previously had randomly initialized weights, meaning the quality of the weights will vary. But, if we were to apply transfer learning by using a larger model that's been trained on a larger dataset to solve similar problem, the CNN would perform better for our specific problem, depite our network being trained on a small dataset. VGG16 ModelThe VGG16 Model is a high-performace, pre-trained model that was developed for image recognition tasks.<jupyter_code># Load the VGG16 model
vgg16 = keras.applications.VGG16(weights='imagenet', include_top=False)
vgg16.summary()
# Use the VGG16 model as a base and add our own layers onto the bottom of it
base_model = vgg16

new_model = base_model.output
new_model=keras.layers.GlobalAveragePooling2D()(new_model)
new_model=keras.layers.Dense(30,activation='relu')(new_model)
new_model=keras.layers.Dense(16,activation='softmax')(new_model)
model=keras.Model(inputs=base_model.input,outputs=new_model)

model.compile(loss=keras.losses.CategoricalCrossentropy(),
              optimizer=keras.optimizers.Adam(),
              metrics=[keras.metrics.CategoricalAccuracy()])

model.summary()
#Train model
batch_size = 20
epochs = 10
history2 = model.fit(x_train, y_train,
                    batch_size = batch_size,
                    epochs = epochs,
                    verbose = 1,
                    validation_split = 0.2)
# Testing
results = model.evaluate(x_test, y_test, verbose=1)
print('Test loss:', results[0])
print('Test accuracy:', results[1])
plt.figure()

# Summarize history for accuracy  
plt.subplot(211)
plt.plot(history2.history['categorical_accuracy'])
plt.plot(history2.history['val_categorical_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['training','validation'],loc='upper left')

# Summarize history for loss  
plt.subplot(212)
plt.plot(history2.history['loss'])
plt.plot(history2.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['training','validation'],loc='upper left')
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>This model had a lot less overfitting compared to the previous model. However, the accuracy was about the same.  Comparing Accuracy and Loss<jupyter_code>plt.figure()

plt.subplot(211)
plt.plot(history.history['val_categorical_accuracy'])
plt.plot(history2.history['val_categorical_accuracy'])
plt.title('validation accuracy')
plt.xlabel('epoch')
plt.ylabel('accuracy')
# Model 1 respresents the simple CNN 
# Model 2 represents the CNN that uses transfer learning from VGG16
plt.legend(['model 1', 'model 2'],loc='upper left')

plt.subplot(212)
plt.plot(history.history['val_loss'])
plt.plot(history2.history['val_loss'])
plt.title('validation loss')
plt.xlabel('epoch')
plt.ylabel('loss')
# Model 1 respresents the simple CNN 
# Model 2 represents the CNN that uses transfer learning from VGG16
plt.legend(['model 1', 'model 2'],loc='upper left')
plt.tight_layout()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08740513345271596, 'fraction_numerical': 0.023705977658395157, 'mean_word_length': 4.855217174238642, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '138460', 'license': 'MIT', 'n_tokens_mistral': 3598, 'n_tokens_neox': 3307, 'n_words': 1281, 'path': 'Demo/Demo.ipynb', 'repo_name': 'CSCI4850/s21-team5-project', 'size': '11727'}