<jupyter_start><jupyter_text>Project: Build a Traffic Sign Recognition Classifier --- Step 0: Load The Data<jupyter_code># Load pickled data
import pickle

# TODO: Fill this in based on where you saved the training and testing data

training_file = "data/train.p"
validation_file= "data/valid.p"
testing_file = "data/test.p"

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']<jupyter_output><empty_output><jupyter_text>--- Step 1: Dataset Summary & Exploration  Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas<jupyter_code>### Replace each question mark with the appropriate value. 
### Use python, pandas or numpy methods rather than hard coding the results
import numpy as np

# TODO: Number of training examples
n_train = len(X_train)

# TODO: Number of validation examples
n_validation = len(X_valid)

# TODO: Number of testing examples.
n_test = len(X_test)

# TODO: What's the shape of an traffic sign image?
image_shape = X_train[0].shape

# TODO: How many unique classes/labels there are in the dataset.
n_classes = len(np.unique(y_train))

print("Number of training examples =", n_train)
print("Number of validating examples =", n_validation)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)<jupyter_output>Number of training examples = 34799
Number of validating examples = 4410
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43<jupyter_text>Include an exploratory visualization of the dataset Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.<jupyter_code>### Data exploration visualization code goes here.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

# Visualizations will be shown in the notebook.
%matplotlib inline
plt.figure(figsize=(18,10))
for i in range (20):
    x_indice = np.random.randint(1, len(X_train))
    x_vis = X_train[x_indice]
    plt.subplot(4,5,1+i)
    plt.imshow(x_vis)
    plt.title(y_train[x_indice])
    plt.axis('off')
# statistic of training data as Visualization Example
from collections import Counter
csv_data = pd.read_csv('signnames.csv')
#csv_data.head()
csv_datan = np.array(csv_data)
label_content=[]
for i in range(len(csv_datan[:,1])):
    label_content.append(str(csv_datan[:,1][i]))

plt.figure(figsize=(16,12))
y_sort = sorted(y_train)
agg = Counter(y_sort)
value_list = []
key_list = []
for key in agg:
    value_list.append(agg[key])
    key_list.append(key)
print(value_list)
print(key_list)
plt.barh(range(len(csv_datan[:,1])), value_list, align = 'center',color= 'steelblue', alpha = 1)
plt.yticks(range(len(csv_datan[:,1])),label_content)
plt.xlabel('Number of each subset')
plt.title('Training Data Distribution')<jupyter_output>[180, 1980, 2010, 1260, 1770, 1650, 360, 1290, 1260, 1320, 1800, 1170, 1890, 1920, 690, 540, 360, 990, 1080, 180, 300, 270, 330, 450, 240, 1350, 540, 210, 480, 240, 390, 690, 210, 599, 360, 1080, 330, 180, 1860, 270, 300, 210, 210]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42]<jupyter_text>---- Step 2: Design and Test a Model Architecture  Pre-process the Data Set<jupyter_code>import matplotlib.pyplot as plt
import cv2
from sklearn.utils import shuffle
### converting to grayscale, etc.
def rgb_gray_change(Dataset):
    X_gray =[]
    for i in range(len(Dataset)):
        X_gray.append(cv2.cvtColor(Dataset[i],cv2.COLOR_RGB2GRAY))
    X_gray_np = np.array(X_gray)
    return X_gray_np

def norm_shape(Dataset):
    X_norm = (Dataset-127.5)/127.5
    return X_norm

def reshape_plus(Dataset):
    X_norm = Dataset.reshape(Dataset.shape + (1,))
    return X_norm

X_train_gr = rgb_gray_change(X_train)
X_valid_gr = rgb_gray_change(X_valid)
X_test_gr = rgb_gray_change(X_test)

plt.figure(figsize=(4,4))
plt.subplot(1,2,1)
plt.imshow(X_train[120])
plt.title('RGB Image')
plt.subplot(1,2,2)
plt.imshow(X_train_gr[120], cmap = 'gray')
plt.title('Gray Image')
X_train_grn = norm_shape(X_train_gr)
X_valid_grn = norm_shape(X_valid_gr)
X_test_grn = norm_shape(X_test_gr)

plt.figure(figsize=(4,4))
plt.subplot(1,2,1)
plt.imshow(X_train_gr[120], cmap = 'gray')
plt.title('Unnormalized Image')
plt.subplot(1,2,2)
plt.imshow(X_train_grn[120], cmap = 'gray')
plt.title('Normalized Image')
X_train_grn, y_train = shuffle(reshape_plus(X_train_grn), y_train)
X_valid_grn, y_valid = shuffle(reshape_plus(X_valid_grn), y_valid)
X_test_grn, y_test = shuffle(reshape_plus(X_test_grn), y_test)

print(X_train_grn.shape, y_train.shape)
print(X_valid_grn.shape, y_valid.shape)<jupyter_output>(34799, 32, 32, 1) (34799,)
(4410, 32, 32, 1) (4410,)<jupyter_text>Model Architecture<jupyter_code>### Define your architecture here.
import tensorflow as tf
from tensorflow.contrib.layers import flatten

def Model(x,keep_prob):    
    mu = 0
    sigma = 0.1
    
    # conv
    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 1, 6), mean = mu, stddev = sigma))
    conv1_b = tf.Variable(tf.zeros(6))
    conv1 = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b
    conv1 = tf.layers.batch_normalization(conv1, training=True)
    conv1 = tf.nn.relu(conv1)

    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))
    conv2_b = tf.Variable(tf.zeros(16))
    conv2 = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
    conv2 = tf.layers.batch_normalization(conv2, training=True)
    conv2 = tf.nn.relu(conv2)
     
    conv3_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 16, 32), mean = mu, stddev = sigma))
    conv3_b = tf.Variable(tf.zeros(32))
    conv3 = tf.nn.conv2d(conv2, conv3_W, strides=[1, 1, 1, 1], padding='VALID') + conv3_b
    conv3 = tf.layers.batch_normalization(conv3, training=True)
    conv3 = tf.nn.relu(conv3)
    conv3 = tf.nn.dropout(conv3, keep_prob)
    
    # full connect
    fc0   = flatten(conv3)
    
    fc0 = tf.layers.batch_normalization(fc0, training=True)
    
    fc1_W = tf.Variable(tf.truncated_normal(shape=(1152, 256), mean = mu, stddev = sigma))
    fc1_b = tf.Variable(tf.zeros(256))
    fc1 = tf.matmul(fc0, fc1_W) + fc1_b
    fc1 = tf.nn.relu(fc1)

    fc2_W  = tf.Variable(tf.truncated_normal(shape=(256, 64), mean = mu, stddev = sigma))
    fc2_b  = tf.Variable(tf.zeros(64))
    fc2 = tf.matmul(fc1, fc2_W) + fc2_b
    fc2 = tf.nn.relu(fc2)

    fc3_W  = tf.Variable(tf.truncated_normal(shape=(64, 43), mean = mu, stddev = sigma))
    fc3_b  = tf.Variable(tf.zeros(43))
    logits = tf.matmul(fc2, fc3_W) + fc3_b
    
    return logits<jupyter_output><empty_output><jupyter_text>Train, Validate and Test the Model A validation set can be used to assess how well the model is performing. A low accuracy on the training and validationsets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.<jupyter_code># create Placeholder
x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)
keep_prob = tf.placeholder(tf.float32)
# training configuration (blank)
logits = Model(x,keep_prob)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = 0.001)
training_operation = optimizer.minimize(loss_operation)

correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()
# validation 
def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y,keep_prob: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples<jupyter_output><empty_output><jupyter_text>Run<jupyter_code>EPOCHS= 50
BATCH_SIZE = 128

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train_grn)
    
    print("Training...")
    print()
    for i in range(EPOCHS):
        acc_train = 0
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train_grn[offset:end], y_train[offset:end]
            acc,loss = sess.run([accuracy_operation,training_operation], feed_dict={x: batch_x, y: batch_y ,keep_prob: 0.5})  
            acc_train += (acc * len(batch_x))
        train_accuracy = acc_train/ num_examples
        validation_accuracy = evaluate(X_valid_grn, y_valid)
        if (i+1) % 5 == 0:
            print("EPOCH {} ...".format(i+1))
            print("Train Accuracy = {:.3f}".format(train_accuracy))
            print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        
    saver.save(sess, './model')
    print("Model saved")
with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    test_accuracy = evaluate(X_test_grn, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))<jupyter_output>INFO:tensorflow:Restoring parameters from ./model
Test Accuracy = 0.951<jupyter_text>--- Step 3: Test a Model on New ImagesTo give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.You may find `signnames.csv` useful as it contains mappings from the class id (integer) to the actual sign name.<jupyter_code>### Load and Output the Images
### Load the images and plot them here.
import glob
import matplotlib.pyplot as plt
import cv2
import numpy as np

plt.figure(figsize=(12,10))
img_extra  = glob.glob('extra-image-test/*')
true_label = [17,12,23,26,1,4,14]
label_ex = []
for k in range(len(img_extra)):
    imgs_ex = cv2.cvtColor(cv2.imread(img_extra[k]),cv2.COLOR_BGR2RGB)
    label_ex.append(label_content[true_label[k]])
    plt.subplot(1,7,1+k)
    plt.imshow(imgs_ex)
    plt.axis('off')
    plt.title(true_label[k])
print(label_ex)
    
### Predict the Sign Type for Each Image

def rgb_gray_exp(Dataset):
    X_gray =[]
    for i in range(len(Dataset)):
        read_image = cv2.imread(Dataset[i])
        resized = cv2.resize(read_image,(32,32))
        X_gray.append(cv2.cvtColor(resized,cv2.COLOR_BGR2GRAY))
    X_gray_np = np.array(X_gray)
    return X_gray_np

extra_gr = rgb_gray_exp(img_extra)
extra_gr = (extra_gr-127.5)/127.5
extra_gr = extra_gr.reshape(len(img_extra),32,32,1)
print(extra_gr.shape)

def predict(features):
    with tf.Session() as sess:
        saver.restore(sess, tf.train.latest_checkpoint('.'))
        results = sess.run(tf.argmax(logits, 1), {x : features,keep_prob: 1.0})
        return results
    
pred = predict(extra_gr) 
print(pred)

plt.figure(figsize=(12,10))
for t in range(len(img_extra)):
    imgs_ex = cv2.cvtColor(cv2.imread(img_extra[t]),cv2.COLOR_BGR2RGB)
    plt.subplot(1,7,1+t)
    plt.imshow(imgs_ex)
    plt.axis('off')
    plt.title(label_content[pred[t]],fontsize=8)<jupyter_output>(7, 32, 32, 1)
INFO:tensorflow:Restoring parameters from ./model
[17 12 23 26  2 16  3]<jupyter_text>Analyze Performance<jupyter_code>### Calculate the accuracy for these 5 new images. 
### For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate on these new images.

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    exp_accuracy = evaluate(extra_gr, true_label)
    print("Test Set Accuracy = {:.3f}".format(exp_accuracy))<jupyter_output>INFO:tensorflow:Restoring parameters from ./model
Test Set Accuracy = 0.571<jupyter_text>Output Top 5 Softmax Probabilities For Each Image Found on the Web<jupyter_code>### Print out the top five softmax probabilities for the predictions on the German traffic sign images found on the web. 
softmax_logits = tf.nn.softmax(logits)
top_k = tf.nn.top_k(softmax_logits, k=5)


with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, tf.train.latest_checkpoint('.'))
    top_5 = sess.run(top_k, feed_dict={x: extra_gr, keep_prob: 1.0})
    soft_5 = sess.run(softmax_logits, feed_dict={x: extra_gr, keep_prob: 1.0})
    top_result_indices = top_5.indices
    top_result_values = top_5.values
    print(top_result_indices)
    print(top_result_values)
for pr in range(len(extra_gr)):
    print('the Top 5 Softmax Probabilities of No. '+ str(pr+1) + ' images is:')
    for index in range(5):
        print(str(label_content[top_result_indices[pr][index]]) + ':' + str("%.2f"%(100 *top_result_values[pr][index])) + '%')
    print()<jupyter_output>the Top 5 Softmax Probabilities of No. 1 images is:
No entry:100.00%
Turn left ahead:0.00%
No passing:0.00%
Speed limit (60km/h):0.00%
Go straight or right:0.00%

the Top 5 Softmax Probabilities of No. 2 images is:
Priority road:100.00%
No passing:0.00%
End of no passing:0.00%
Speed limit (100km/h):0.00%
Ahead only:0.00%

the Top 5 Softmax Probabilities of No. 3 images is:
Slippery road:100.00%
No passing for vehicles over 3.5 metric tons:0.00%
Turn left ahead:0.00%
Speed limit (60km/h):0.00%
Beware of ice/snow:0.00%

the Top 5 Softmax Probabilities of No. 4 images is:
Traffic signals:99.99%
Bicycles crossing:0.01%
General caution:0.00%
Children crossing:0.00%
Speed limit (120km/h):0.00%

the Top 5 Softmax Probabilities of No. 5 images is:
Speed limit (50km/h):99.74%
Speed limit (30km/h):0.26%
Speed limit (80km/h):0.00%
Speed limit (60km/h):0.00%
No vehicles:0.00%

the Top 5 Softmax Probabilities of No. 6 images is:
Vehicles over 3.5 metric tons prohibited:72.33%
Dangerous curve to the[...]
{'fraction_non_alphanumeric': 0.130111011514859, 'fraction_numerical': 0.050610218575467145, 'mean_word_length': 4.5892100192678225, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '398711', 'license': 'MIT', 'n_tokens_mistral': 5926, 'n_tokens_neox': 5351, 'n_words': 1334, 'path': 'Traffic_Sign_Classifier.ipynb', 'repo_name': 'ChenweiSun/SDND-Traffic-Sign-Classification', 'size': '14503'}