<jupyter_start><jupyter_text>Self-Driving Car Engineer Nanodegree Deep Learning Project: Build a Traffic Sign Recognition Classifier --- Import Libraries<jupyter_code># Import libraries
import pickle
import pandas as pd
import numpy as np
import random
import cv2
import os

# Import Matlab
import matplotlib.pyplot as plt
%matplotlib inline

# Import tensorflow and use version 1
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()

from tensorflow.compat.v1.layers import flatten
from sklearn.utils import shuffle

# Set random seed
random.seed(10)<jupyter_output>WARNING:tensorflow:From d:\selfdrivingcars\car\lib\site-packages\tensorflow\python\compat\v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term<jupyter_text>--- Load The Data<jupyter_code># Load pickled data
training_file = 'data/train.p'
validation_file = 'data/valid.p'
testing_file = 'data/test.p'
signnames_file = 'signnames.csv'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']
# Load signnames
signnames = pd.read_csv(signnames_file, index_col = 0)<jupyter_output><empty_output><jupyter_text>--- Step 1: Dataset Summary & ExplorationThe pickled data is a dictionary with 4 key/value pairs:- `'features'` is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).- `'labels'` is a 1D array containing the label/class id of the traffic sign. The file `signnames.csv` contains id -> name mappings for each id.- `'sizes'` is a list containing tuples, (width, height) representing the original width and height the image.- `'coords'` is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. **THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES**   Dataset Summary<jupyter_code># TODO: Number of training examples
n_train = len(X_train)

# TODO: Number of validation examples
n_validation = len(X_valid)

# TODO: Number of testing examples.
n_test = len(X_test)

# TODO: What's the shape of an traffic sign image?
image_shape = X_train[0].shape

# TODO: How many unique classes/labels there are in the dataset.
n_classes = len(np.unique(
                np.concatenate([y_train, y_valid, y_test])
            ))

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Number of validation examples =", n_validation)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)<jupyter_output>Number of training examples = 34799
Number of testing examples = 12630
Number of validation examples = 4410
Image data shape = (32, 32, 3)
Number of classes = 43<jupyter_text>Dataset Exploration<jupyter_code># View some random images
vis_n = 5
fig, axes = plt.subplots(1, vis_n, figsize=(20, 5), squeeze = True)
random_indices = random.choices(range(len(X_train)), k = vis_n)

for i in range(vis_n):
    index = random_indices[i]
    axes[i].imshow(X_train[index])
    axes[i].set_title(signnames.iloc[y_train[index]]["SignName"])
# Data Distribution of Training Data
unique = np.unique(y_train)

fig, ax = plt.subplots(figsize =(10, 7))
ax.hist(y_train, bins = unique, rwidth = 0.5)
ax.set_xlim([0, 42])
# Data Distribution of Validation Data
unique = np.unique(y_valid)

fig, ax = plt.subplots(figsize =(10, 7))
ax.hist(y_valid, bins = unique, rwidth = 0.5)
ax.set_xlim([0, 42])
# Data Distribution of Testing Data
unique = np.unique(y_test)

fig, ax = plt.subplots(figsize =(10, 7))
ax.hist(y_test, bins = unique, rwidth = 0.5)
ax.set_xlim([0, 42])<jupyter_output><empty_output><jupyter_text>---- Design and Test a Model Architecture  Pre-process the Data Set (normalization and grayscale)<jupyter_code># Function to convert to grayscale
def convert_grayscale(X):
    X_gray = np.zeros((X.shape[0], 32, 32, 1))
    for i in range(X.shape[0]):
        X_gray[i] = cv2.cvtColor(X[i], cv2.COLOR_RGB2GRAY).reshape((32, 32, -1))
        
    return X_gray

# Convert image to grayscale
X_train = convert_grayscale(X_train)
X_valid = convert_grayscale(X_valid)
X_test = convert_grayscale(X_test)
# View grayscale images
vis_n = 5
fig, axes = plt.subplots(1, vis_n, figsize=(20, 5), squeeze = True)

for i in range(vis_n):
    index = random_indices[i]
    axes[i].imshow(X_train[index], cmap = 'Greys')
    axes[i].set_title(signnames.iloc[y_train[index]]["SignName"])
# Normalize the data
X_train = (X_train - 128) / 128
X_valid = (X_valid - 128) / 128
X_test = (X_test - 128) / 128

X_train = np.float32(X_train)
X_valid = np.float32(X_valid)
X_test = np.float32(X_test)<jupyter_output><empty_output><jupyter_text>Model Architecture<jupyter_code># Define input and output
x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)

# Define dropout_probability
dropout_prob = tf.placeholder(tf.float32)
# Function for convolutional layer
def convolution_layer(x, shape, mu = 0, sigma = 0.1):
    layer_W = tf.Variable(tf.truncated_normal(shape = shape, mean = mu, stddev = sigma))
    layer_b = tf.Variable(tf.zeros(shape[-1]))
    
    layer = tf.nn.conv2d(x, layer_W, strides = [1, 1, 1, 1], padding = 'VALID') + layer_b
    layer = tf.nn.relu(layer)
    layer = tf.nn.dropout(layer, dropout_prob)
    layer = tf.nn.max_pool(layer, ksize = [1, 2, 2, 1], strides = [1, 2, 2, 1], padding = 'VALID')
    
    return layer

# Function for fully connected layer
def fully_connected_layer(x, shape, mu = 0, sigma = 0.1, activation = True):
    layer_W = tf.Variable(tf.truncated_normal(shape = shape, mean = mu, stddev = sigma))
    layer_b = tf.Variable(tf.zeros(shape[-1]))
    layer = tf.matmul(x, layer_W) + layer_b
    layer = tf.nn.dropout(layer, dropout_prob)
    
    if activation:
        layer = tf.nn.relu(layer)
        
    return layer

# Function for LeNet network
def network(x):
    mu = 0
    sigma = 0.1
    
    # Layer 1
    layer1 = convolution_layer(x, (5, 5, 1, 10), mu, sigma)
    
    # Layer 2
    layer2 = convolution_layer(layer1, (5, 5, 10, 32), mu, sigma)
    
    # Flat Layer 0
    fclayer0 = flatten(layer2)
    
    # Flat Layer 1
    fclayer1 = fully_connected_layer(fclayer0, (800, 500), mu, sigma)
    
    # Flat Layer 2
    fclayer2 = fully_connected_layer(fclayer1, (500, 200), mu, sigma)
    
    # Flat Layer 3
    fclayer3 = fully_connected_layer(fclayer2, (200, 100), mu, sigma)
    
    # Flat Layer 4
    fclayer4 = fully_connected_layer(fclayer3, (100, 43), mu, sigma, False)
    
    return fclayer4<jupyter_output><empty_output><jupyter_text>Train, Validate and Test the Model<jupyter_code># Training parameters
BATCH_SIZE = 16
EPOCHS = 30
RATE = 0.001
DROPOUT_PROB = 0.6
# Define the training pipeline
logits = network(x)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits = logits, labels = one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = RATE)
training_operation = optimizer.minimize(loss_operation)
# Define evaluation pipeline
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Function to evaluate an input and output
def evaluate(X_evaluate, y_evaluate):
    number_examples = len(X_evaluate)
    total_accuracy = 0
    session = tf.get_default_session()
    for offset in range(0, number_examples, BATCH_SIZE):
        batch_x, batch_y = X_evaluate[offset : offset + BATCH_SIZE], y_evaluate[offset : offset + BATCH_SIZE]
        accuracy = session.run(accuracy_operation, feed_dict = {x: batch_x, y: batch_y, dropout_prob: 1})
        total_accuracy += accuracy * len(batch_x)
        
    return total_accuracy / number_examples
# Training process
with tf.Session() as session:
    session.run(tf.global_variables_initializer())
    number_examples = len(X_train)
    
    print("Training ...")
    print()
    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        
        # Train batches of images
        for offset in range(0, number_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset : end], y_train[offset : end]
            session.run(training_operation, feed_dict = {x: batch_x, y: batch_y, dropout_prob: DROPOUT_PROB})
         
        print(f"EPOCH {i + 1} ...")
        training_accuracy = evaluate(X_train, y_train)
        validation_accuracy = evaluate(X_valid, y_valid)
        print("Training Accuracy {:.3f}".format(training_accuracy))
        print("Validation Accuracy {:.3f}".format(validation_accuracy))
        print()
        
    try:
        saver
    except NameError:
        saver = tf.train.Saver()
        
    # Save the model
    saver.save(session, 'TrafficSign')
    print("Model Saved")
# Evaluating Testing Accuracy
with tf.Session() as session:
    saver.restore(session, 'TrafficSign')
    testing_accuracy = evaluate(X_test, y_test)
    print("Testing Accuracy {:.3f}".format(testing_accuracy))<jupyter_output>INFO:tensorflow:Restoring parameters from TrafficSign
Testing Accuracy 0.934<jupyter_text>--- Test a Model on New Images  Load and Visualize the Images<jupyter_code># Load images
test_image_dir = os.listdir('test')
test_images = []

# Convert from BGR to RGB
for image in test_image_dir:
    bgr_image = cv2.imread('test/' + image)
    rgb_image = cv2.cvtColor(bgr_image, cv2.COLOR_BGR2RGB)
    test_images.append(rgb_image)
# Visualize the images
vis_n = 5
fig, axes = plt.subplots(1, vis_n, figsize=(20, 5), squeeze = True)

for i in range(vis_n):
    axes[i].imshow(test_images[i])<jupyter_output><empty_output><jupyter_text>Predict the Sign Type for Each Image<jupyter_code># Convert images for input to network
tf_test_images = np.zeros((5, 32, 32, 1))

for index in range(len(test_images)):
    gray_image = cv2.cvtColor(test_images[index], cv2.COLOR_RGB2GRAY)
    resize_image = cv2.resize(gray_image, (32, 32))
    tf_test_images[index] = resize_image.reshape((32, 32, -1))
    tf_test_images[index] = (tf_test_images[index] - 128) / 128
# Test model on new images
test_prediction = tf.argmax(logits, 1)

with tf.Session() as session:
    saver.restore(session, 'TrafficSign')
    predictions = session.run(test_prediction, feed_dict = {x: tf_test_images, dropout_prob: 1})
# Print the predictions
for i in range(5):
    prediction = predictions[i]
    predicted_sign = signnames.iloc[prediction]["SignName"]
    print(f"Prediction for image {i} is {predicted_sign}")<jupyter_output>Prediction for image 0 is Children crossing
Prediction for image 1 is Road work
Prediction for image 2 is Right-of-way at the next intersection
Prediction for image 3 is Stop
Prediction for image 4 is Speed limit (80km/h)<jupyter_text>Analyze Performance<jupyter_code>print(f"The accuracy is {(4/5) * 100} %")
print(f"The last prediction should have been Speed limit (60 km/h)")<jupyter_output>The accuracy is 80.0 %
The last prediction should have been Speed limit (60 km/h)<jupyter_text>Output Top 5 Softmax Probabilities For Each Image Found on the Web<jupyter_code>k = 5
top_k_predictions = tf.nn.top_k(logits, k = k)
softmax_predictions = tf.nn.softmax(top_k_predictions[0])

# Get the top 5 probabilities
with tf.Session() as session:
    saver.restore(session, 'TrafficSign')
    top_logits = session.run(top_k_predictions, feed_dict = {x : tf_test_images, dropout_prob: 1})
    top_probabilities = session.run(softmax_predictions, feed_dict = {x: tf_test_images, dropout_prob: 1})
    
# Print the probabilities
for i in range(5):
    print(f"Image Predictions for {i}")
    for j in range(k):
        predicted_sign = signnames.iloc[top_logits[1][i][j]]["SignName"]
        probability = top_probabilities[i][j]
        print(f"Prediction {j}\t{probability}\t{predicted_sign}")
    print()<jupyter_output>INFO:tensorflow:Restoring parameters from TrafficSign
Image Predictions for 0
Prediction 0	0.5528236627578735	Children crossing
Prediction 1	0.1495007872581482	Beware of ice/snow
Prediction 2	0.11569516360759735	Slippery road
Prediction 3	0.09443562477827072	Bicycles crossing
Prediction 4	0.087544746696949	Right-of-way at the next intersection

Image Predictions for 1
Prediction 0	0.2706294059753418	Road work
Prediction 1	0.22730791568756104	Speed limit (80km/h)
Prediction 2	0.18013347685337067	Beware of ice/snow
Prediction 3	0.16409888863563538	Speed limit (100km/h)
Prediction 4	0.1578303724527359	Speed limit (120km/h)

Image Predictions for 2
Prediction 0	0.9871540665626526	Right-of-way at the next intersection
Prediction 1	0.011918905191123486	Beware of ice/snow
Prediction 2	0.0005791957955807447	Double curve
Prediction 3	0.00025007969816215336	Pedestrians
Prediction 4	9.77448871708475e-05	Roundabout mandatory

Image Predictions for 3
Prediction 0	1.0	Stop
Prediction 1	6.99404617124[...]
{'fraction_non_alphanumeric': 0.10837143071273327, 'fraction_numerical': 0.04579180094431537, 'mean_word_length': 4.466612044244163, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '394067', 'license': 'MIT', 'n_tokens_mistral': 5110, 'n_tokens_neox': 4627, 'n_words': 1259, 'path': 'Traffic_Sign_Classifier.ipynb', 'repo_name': 'SakshayMahna/P3-TrafficSignClassifier', 'size': '13343'}