<jupyter_start><jupyter_text>2228 CougarTech Galactic Search Neural Network OverviewThis notebook will compile and train a MobileNetV2 based Convolutional Neural Network (CNN) to detect the 4 possible field layouts for the 2021 Galactic Search autonomous challange.Using training images captured of all the various field layouts from different possible robot angles and perspectives, the model will be trained and saved off to a Tensorflow .pb file that can be loaded onto the Raspberry Pi and opened up using OpenCV 4.5.1 Training the ModelAll of the training, validation, and test images must be in the following directory structure:```  - data   `- GalacticSearch     |- test     | |- BlueA     | |- BlueB     | |- RedA     | `- RedB     |- train     | |- BlueA     | |- BlueB     | |- RedA     | `- RedB     `- valid       |- BlueA       |- BlueB       |- RedA       `- RedB```- **Train** images are the actual images fed into the network durring it's training process. The more images you have here, the more reliable your network will end up.- **Valid** images are the images used for network validation durring the training process. These images should be separate from the training images, as the point is to feed the network images it has never seen before to determine how well it can gerneralze input.- **Test** images are like the validation images, but a smaller sub-set used to perform one final test of the network to determine how well it performsTo perform the actual model training, simply run this notebook. It will result in the compiled model being written to the `models` folder DeployingThe Raspberry Pi wpilibpi vision application must be set to "custom" instead of "uploaded Java jar" or any other option. The actual copying of the vision code, and the required libraries and CNN model will be performed by gradle.Note: The stock wpilibpi image only supports OpenCV 3.4.7, so a custom build of 4.5.1 was made, and will be deployed to the Pi automaticallyTo deploy all required components to the Pi, and automatically restart the vision service, run the following commands from VSCode:- `./gradlew build` -- Build the actual Java application- `./gradlew deploy` -- Deploy everything to the PiThis will do the following:- Check for the presence of `/home/pi/opencv-4.5.1`. If this does not exist, it will copy over the opencv-4.5.1 distribution and extract it to disk- Check for the presence of the CNN .pb file, if not found, it will copy it to `/home/pi/`- Copy the latest compiled .jar file, and the runCamera command to `/home/pi/`- Restart the camera service, causing the updated code to start running<jupyter_code># Top-Level imports and definition of some helper fuction

import numpy as np
import tensorflow as tf
from tensorflow import keras
import tensorflow.keras.backend
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Activation, Dense, Flatten, BatchNormalization, Conv2D, MaxPool2D, GlobalAveragePooling2D
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.python.framework.convert_to_constants import convert_variables_to_constants_v2
from sklearn.metrics import confusion_matrix
import itertools
import os
import shutil
import random
import glob
import matplotlib.pyplot as plt
import warnings
import ssl

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    pass
else:
    ssl._create_default_https_context = _create_unverified_https_context
    
warnings.simplefilter(action='ignore', category=FutureWarning)
%matplotlib inline

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
            horizontalalignment="center",
            color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    
def plotImages(images_arr):
    fig, axes = plt.subplots(1, 10, figsize=(20,20))
    axes = axes.flatten()
    for img, ax in zip( images_arr, axes):
        ax.imshow(img)
        ax.axis('off')
    plt.tight_layout()
    plt.show()
# Load up the training data
train_path = 'data/GalacticSearch/train'
valid_path = 'data/GalacticSearch/valid'
test_path = 'data/GalacticSearch/test'
height=224
width=224

train_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet_v2.preprocess_input) \
    .flow_from_directory(directory=train_path, target_size=(width,height), classes=['RedA', 'RedB', 'BlueA', 'BlueB'], batch_size=10)
valid_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet_v2.preprocess_input) \
    .flow_from_directory(directory=valid_path, target_size=(width,height), classes=['RedA', 'RedB', 'BlueA', 'BlueB'], batch_size=10)
test_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet_v2.preprocess_input) \
    .flow_from_directory(directory=test_path, target_size=(width,height), classes=['RedA', 'RedB', 'BlueA', 'BlueB'], batch_size=10, shuffle=False)
# Quick sanity check to make sure our training data is what we think it should be
imgs, labels = next(train_batches)
plotImages(imgs)
print(labels)
# Define and compile the model.
# We are using the pre-defined MobileNetV2 model as it's a well-established model
# That is designed to run on mobile devices with less-power CPUs, like the
# Raspberry Pi

# MobileNetV2 is defined as taking in a 224x224x3 RGB Matrix, and classifying
# the image into one of 1000 caregories. As we only need 4 gategories for
# Galactic search, we'll strip off the last 1000-node clasification layer,
# and add in a few more layers ending with a softmax classification layer for
# 4 labels.

# Imports the MobileNetV2 model and discards the last 1000 neuron layer.
base_model = tf.keras.applications.MobileNetV2(
    weights='imagenet',include_top=False) 
x=base_model.output
x=GlobalAveragePooling2D()(x)

# we add dense layers so that the model can learn more complex functions
# and classify for better results.
x=Dense(1024,activation='relu')(x)

# dense layer 2
x=Dense(1024,activation='relu')(x)

# dense layer 3
x=Dense(512,activation='relu')(x)

# final layer with softmax activation for 4 classes
preds=Dense(4,activation='softmax')(x) 

# specify the inputs and outputs
model=Model(inputs=base_model.input,outputs=preds)

# Mark the first 20 layers as Frozen, as we want to use
# what the default model already knows
for layer in model.layers[:20]:
    layer.trainable=False
for layer in model.layers[20:]:
    layer.trainable=True

# print it out so we can see what we're dealing with
model.summary()

# we need to compile the model before we can use it. We're using pretty
# standard optimizers and loss calculation methods here
model.compile(optimizer=Adam(learning_rate=0.0001), loss='categorical_crossentropy', metrics=['accuracy'])
# Do the actual learning step
model.fit(x=train_batches,
          steps_per_epoch=len(train_batches),
          validation_data=valid_batches,
          validation_steps=len(valid_batches),
          epochs=20,
          verbose=2
)
# Run the test data through and plot the results

test_imgs, test_labels = next(test_batches)
# plotImages(test_imgs)
# print(test_labels)

# Do the actual predictions
predictions = model.predict(x=test_batches, steps=len(test_batches), verbose=0)

print (predictions)
# round the probabilities to 0-1
np.round(predictions)

# print out the indcies of each class label in the test data
# we need to make sure the order here matches the order in the
# cm_plot_labels below
test_batches.class_indices

cm = confusion_matrix(y_true=test_batches.classes, y_pred=np.argmax(predictions, axis=-1))

cm_plot_labels = ['RedA','RedB', 'BlueA', 'BlueB']
plot_confusion_matrix(cm=cm, classes=cm_plot_labels, title='Confusion Matrix')

# This cell will write out the trained model in a tensorflow format (.pb)
# so that it can be read in by OpenCV in the actual vision code
# running on the Pi

# Directory name to save the model to
frozen_out_path = 'models'

# Name of the .pb file
frozen_graph_filename = 'galactic_search'

# Convert Keras model to ConcreteFunction
full_model = tf.function(lambda x: model(x))
full_model = full_model.get_concrete_function(
    tf.TensorSpec(model.inputs[0].shape, model.inputs[0].dtype))

frozen_func = convert_variables_to_constants_v2(full_model)
frozen_func.graph.as_graph_def()

tf.io.write_graph(graph_or_graph_def=frozen_func.graph,
                  logdir=frozen_out_path,
                  name=f"{frozen_graph_filename}.pb",
                  as_text=False)

# tf.io.write_graph(graph_or_graph_def=frozen_func.graph,
#                   logdir=frozen_out_path,
#                   name=f"{frozen_graph_filename}.pbtxt",
#                   as_text=True)

# Also save a keras formatted version of the model so that we
# can re-import it here if we want later
model.save(os.path.join(frozen_out_path, frozen_graph_filename + ".h5"))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08523189585028479, 'fraction_numerical': 0.012205044751830757, 'mean_word_length': 4.25267094017094, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '111630', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3031, 'n_tokens_neox': 2881, 'n_words': 1016, 'path': 'neuralNetwork/GalacticSearch-MobileNetV2.ipynb', 'repo_name': 'cougarTech2228/GameChangersVision', 'size': '9832'}