<jupyter_start><jupyter_text>Self-Driving Car Engineer Nanodegree Deep Learning Project: Build a Traffic Sign Recognition ClassifierIn this notebook, a template is provided for you to implement your functionality in stages, which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission if necessary. > **Note**: Once you have completed all of the code implementations, you need to finalize your work by exporting the iPython Notebook as an HTML document. Before exporting the notebook to html, all of the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to  \n",    "**File -> Download as -> HTML (.html)**. Include the finished document along with this notebook as your submission. In addition to implementing code, there is a writeup to complete. The writeup should be completed in a separate file, which can be either a markdown file or a pdf document. There is a [write up template](https://github.com/udacity/CarND-Traffic-Sign-Classifier-Project/blob/master/writeup_template.md) that can be used to guide the writing process. Completing the code template and writeup template will cover all of the [rubric points](https://review.udacity.com/!/rubrics/481/view) for this project.The [rubric](https://review.udacity.com/!/rubrics/481/view) contains "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. The stand out suggestions are optional. If you decide to pursue the "stand out suggestions", you can include the code in this Ipython notebook and also discuss the results in the writeup file.>**Note:** Code and Markdown cells can be executed using the **Shift + Enter** keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
import cv2<jupyter_output><empty_output><jupyter_text>--- Step 0: Load The Data<jupyter_code># Load pickled data
import pickle

# TODO: Fill this in based on where you saved the training and testing data

training_file = 'train.p'
validation_file ='valid.p'
testing_file = 'test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(validation_file, mode='rb') as f:
    valid = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']
X_test, y_test = test['features'], test['labels']
print(X_train[0].dtype)
# Shuffle the training data
from sklearn.utils import shuffle

X_train, y_train = shuffle(X_train, y_train)<jupyter_output><empty_output><jupyter_text>--- Step 1: Dataset Summary & ExplorationThe pickled data is a dictionary with 4 key/value pairs:- `'features'` is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).- `'labels'` is a 1D array containing the label/class id of the traffic sign. The file `signnames.csv` contains id -> name mappings for each id.- `'sizes'` is a list containing tuples, (width, height) representing the original width and height the image.- `'coords'` is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. **THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES**Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the [pandas shape method](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.shape.html) might be useful for calculating some of the summary results.   Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas<jupyter_code>### Replace each question mark with the appropriate value. 
### Use python, pandas or numpy methods rather than hard coding the results

# TODO: Number of training examples
n_train = X_train.shape[0]

# TODO: Number of validation examples
n_validation = X_valid.shape[0]

# TODO: Number of testing examples.
n_test = X_test.shape[0]

# TODO: What's the shape of an traffic sign image?
image_shape = X_train[0].shape

# TODO: How many unique classes/labels there are in the dataset.
n_classes = len(set(y_train))

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)<jupyter_output>Number of training examples = 34799
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43<jupyter_text>Include an exploratory visualization of the dataset Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc. The [Matplotlib](http://matplotlib.org/) [examples](http://matplotlib.org/examples/index.html) and [gallery](http://matplotlib.org/gallery.html) pages are a great resource for doing visualizations in Python.**NOTE:** It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections. It can be interesting to look at the distribution of classes in the training, validation and test set. Is the distribution the same? Are there more examples of some classes than others?<jupyter_code>### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
# Visualizations will be shown in the notebook.
%matplotlib inline
first_index_labels_valid = [np.where(y_valid == label)[0][0] for label in set(y_valid)]
label_images = [X_valid[idx] for idx in first_index_labels_valid]
plt.subplots(6, 7, figsize=(16, 16))
for idx, img, in enumerate(label_images[:-1]):
    plt.subplot(6, 7, idx+1)
    plt.imshow(img)
plt.show()<jupyter_output><empty_output><jupyter_text>Inspect the distributions of Labels  Training Data<jupyter_code>train_hist, bin_edges = np.histogram(y_train, bins = range(n_classes + 1))
plt.bar(bin_edges[:-1], train_hist, width=1)
plt.xlim(min(bin_edges), max(bin_edges))
plt.show()<jupyter_output><empty_output><jupyter_text>Validation data classes<jupyter_code>valid_hist, bin_edges = np.histogram(y_valid, bins = range(n_classes + 1))
plt.bar(bin_edges[:-1], valid_hist, width=1)
plt.xlim(min(bin_edges), max(bin_edges))
plt.show()<jupyter_output><empty_output><jupyter_text>Clearly the data is unevenly distributed. This would cause the more frequently appearing classes to be learnt well as opposed to the ones low in number. There are 2 ways to solve this problem:* Discard all the class samples appearing more in number to 'downsample' to the ```min(hist)``` - which seems silly, since we would be discarding a lot of useful samples just to make it evenly distributed* The more robust but difficult one would be to generate fabricated data to boost the numbers of low numbered classes*Additional data will be augmented after converting images to grayscale below* ---- Step 2: Design and Test a Model ArchitectureDesign and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the [German Traffic Sign Dataset](http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset).The LeNet-5 implementation shown in the [classroom](https://classroom.udacity.com/nanodegrees/nd013/parts/fbf77062-5703-404e-b60c-95b78b2f3f9e/modules/6df7ae49-c61c-4bb2-a23e-6527e69209ec/lessons/601ae704-1035-4287-8b11-e2c2716217ad/concepts/d4aca031-508f-4e0b-b493-e7b706120f81) at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play! With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission. There are various aspects to consider when thinking about this problem:- Neural network architecture (is the network over or underfitting?)- Play around preprocessing techniques (normalization, rgb to grayscale, etc)- Number of examples per label (some have more than others).- Generate fake data.Here is an example of a [published baseline model on this problem](http://yann.lecun.com/exdb/publis/pdf/sermanet-ijcnn-11.pdf). It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.  Pre-process the Data Set (normalization, grayscale, etc.)  1. Grayscale the images<jupyter_code>### Preprocess the data here. It is required to normalize the data. Other preprocessing steps could include 
### converting to grayscale, etc.
### Feel free to use as many code cells as needed.

def grayscale(images):
    return [cv2.cvtColor(img, cv2.COLOR_RGB2GRAY) for img in images]


X_train_gray = grayscale(X_train)
X_valid_gray = grayscale(X_valid)
X_test_gray = grayscale(X_test)
print(X_train_gray[0].shape)<jupyter_output>(32, 32)<jupyter_text>Data manipulation functions that will be used to generate augmentd data<jupyter_code>import random


def random_rotate(img):
    '''Rotate :img: by anti-clockwise direction'''
    degrees = random.randint(1, 25)
    (h, w) = img.shape[:2]
    center = (w / 2, h / 2) 
    M = cv2.getRotationMatrix2D(center, degrees, 1.0)
    return cv2.warpAffine(img, M, (w, h))


def random_translate(img):
    rows,cols = img.shape[:2]
    shift_range = 5
    offset = random.randint(-shift_range, shift_range)
    M = np.float32([[1, 0, offset], [0, 1, offset]])
    return cv2.warpAffine(img,M,(cols,rows))


def random_brightness(img):
    shifted = img + 1.0
    max_value = max(shifted.flatten())
    max_ = 2.0/max_value
    min_ = max_ - 0.1
    coef = np.random.uniform(min_, max_)
    return shifted * coef - 1.0<jupyter_output><empty_output><jupyter_text>Plotting some random images to compare the effects of data manipulation. Top row is original. Bottom is modified<jupyter_code>n_random_imgs = 7

def plot_augmented_data(aug_func, n_imgs):
    original_imgs = [X_train_gray[random.randint(1, 10000)] for _ in range(n_random_imgs)]
    modified_imgs = [aug_func(img) for img in original_imgs]
    plt.subplots(2, n_imgs, figsize=(16, 5))
    plt.suptitle('%s applied' % (aug_func.__name__))
    for idx, img in enumerate(original_imgs):
        plt.subplot(2, n_imgs, idx + 1)
        plt.imshow(img, cmap='gray')
    for idx, img in enumerate(modified_imgs):
        plt.subplot(2, n_imgs, len(original_imgs) + idx + 1)
        plt.imshow(img, cmap='gray')
    
    plt.show()
plot_augmented_data(random_rotate, n_random_imgs)
plot_augmented_data(random_brightness, n_random_imgs)
plot_augmented_data(random_translate, n_random_imgs)<jupyter_output><empty_output><jupyter_text>2. Normalize the data to have zero mean<jupyter_code>def normalize(images):
    images = [img.reshape(32, 32, 1) for img in images]
    lst = [(img - 128.0) / 128 for img in images]
    return np.asarray(lst).reshape(len(lst), 32, 32, 1)
X_train = normalize(X_train_gray)
X_valid = normalize(X_valid_gray)
X_test = normalize(X_test_gray)<jupyter_output><empty_output><jupyter_text>Data AugmentationFor each dataset, we want to boost the minority classes' count upto the mean of counts of all classes<jupyter_code>def augment_data(data, labels):
    hist, _ = np.histogram(labels, bins = range(n_classes + 1))
    boost_limit = int(np.mean(hist))
    minorities = [(idx, count) for idx, count in enumerate(hist) if count < boost_limit]
    print(minorities)
    for cls, count in minorities:
        print('%s with count % started' % (cls, count) + '~' * 10)
        rem = boost_limit - count
        for _ in range(rem):
            aug_func = random.choice([random_rotate, random_translate, random_brightness])
            cls_data = np.where(labels == cls)
            rand_img = data[random.choice(cls_data[0])]
            data = np.append(data, [aug_func(rand_img).reshape(32, 32, 1)], axis=0)
            labels = np.append(labels, [cls])
    return data, labels
X_valid, y_valid = augment_data(X_valid, y_valid)
X_train, y_train = augment_data(X_train, y_train)<jupyter_output>[(0, 30), (6, 60), (14, 90), (15, 90), (16, 60), (19, 30), (20, 60), (21, 60), (22, 60), (23, 60), (24, 30), (26, 60), (27, 30), (28, 60), (29, 30), (30, 60), (31, 90), (32, 30), (33, 90), (34, 60), (36, 60), (37, 30), (39, 30), (40, 60), (41, 30), (42, 30)]
0 with count 30tarted~~~~~~~~~~
6 with count 60tarted~~~~~~~~~~
14 with count 90tarted~~~~~~~~~~
15 with count 90tarted~~~~~~~~~~
16 with count 60tarted~~~~~~~~~~
19 with count 30tarted~~~~~~~~~~
20 with count 60tarted~~~~~~~~~~
21 with count 60tarted~~~~~~~~~~
22 with count 60tarted~~~~~~~~~~
23 with count 60tarted~~~~~~~~~~
24 with count 30tarted~~~~~~~~~~
26 with count 60tarted~~~~~~~~~~
27 with count 30tarted~~~~~~~~~~
28 with count 60tarted~~~~~~~~~~
29 with count 30tarted~~~~~~~~~~
30 with count 60tarted~~~~~~~~~~
31 with count 90tarted~~~~~~~~~~
32 with count 30tarted~~~~~~~~~~
33 with count 90tarted~~~~~~~~~~
34 with count 60tarted~~~~~~~~~~
36 with count 60tarted~~~~~~~~~~
37 with count 30tarted~~~~~~~~~~
39 with count 30t[...]<jupyter_text>Pickle the augmented data<jupyter_code>train_data = {'features': X_train, 'labels': y_train}
valid_data = {'features': X_valid, 'labels': y_valid}

pickle.dump(train_data, open( "augmented_train.p", "wb" ))
pickle.dump(valid_data, open( "augmented_valid.p", "wb" ))<jupyter_output><empty_output><jupyter_text>Load the augmented data<jupyter_code>with open("augmented_train.p", mode='rb') as f:
    train = pickle.load(f)
with open("augmented_valid.p", mode='rb') as f:
    valid = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_valid, y_valid = valid['features'], valid['labels']<jupyter_output><empty_output><jupyter_text>Plotting the histogram disributions of various classes after data augmentation<jupyter_code>valid_hist, bin_edges = np.histogram(y_train, bins = range(n_classes + 1))
plt.bar(bin_edges[:-1], valid_hist, width=1)
plt.xlim(min(bin_edges), max(bin_edges))
plt.title('Augmented training data')
plt.show()
valid_hist, bin_edges = np.histogram(y_valid, bins = range(n_classes + 1))
plt.bar(bin_edges[:-1], valid_hist, width=1)
plt.xlim(min(bin_edges), max(bin_edges))
plt.title('Augmented Validation data')
plt.show()
#Helper function
from tensorflow.contrib.layers import flatten

def get_conv_filter_bias(input_, height, width, depth, mu=0, sigma=0.1):
    input_channels = input_.get_shape()[-1].value
    return (
        tf.Variable(tf.truncated_normal((height, width, input_channels, depth), mean=mu, stddev=sigma)),
        tf.zeros((depth))
    )
# Define model classifier using conv net


def lecun_article_model(x):  # 32 x 32 x 1 
    fw1, b1 = get_conv_filter_bias(x, 5, 5, 6)  # 28 X 28 X 6, 6
    conv_layer1 = tf.nn.bias_add(tf.nn.conv2d(x, fw1, [1, 1, 1, 1], 'VALID'), b1)
    activated1 = tf.nn.relu(conv_layer1)
    activated1 = tf.nn.dropout(activated1, conv_keep_prob)
    pooled1 = tf.nn.max_pool(activated1, [1, 2, 2, 1], [1, 2, 2, 1], 'VALID')  # 14 x 14 x 32
    
    fw2, b2 = get_conv_filter_bias(pooled1, 5, 5, 16)  # 10 x 10 x 16, 16
    conv_layer2 = tf.nn.bias_add(tf.nn.conv2d(pooled1, fw2, [1, 1, 1, 1], 'VALID'), b2)
    activated2 = tf.nn.relu(conv_layer2)
    activated2 = tf.nn.dropout(activated2, conv_keep_prob)
    pooled2 = tf.nn.max_pool(activated2, [1, 2, 2, 1], [1, 2, 2, 1], 'VALID')  # 5 x 5 x 16
    
    fw3, b3 = get_conv_filter_bias(pooled2, 5, 5, 400)  # 1 x 1 x 400, 400
    conv_layer3 = tf.nn.bias_add(tf.nn.conv2d(pooled2, fw3, [1, 1, 1, 1], 'VALID'), b3)
    activated3 = tf.nn.relu(conv_layer3)
    activated3 = tf.nn.dropout(activated3, conv_keep_prob)
    #pooled3 = tf.nn.max_pool(activated3, [1, 2, 2, 1], [1, 2, 2, 1], 'VALID')  # 1 x 1 x 512
    
    '''fw4, b4 = get_conv_filter_bias(pooled3, 2, 2, 400)  # 1 x 1 x 400, 400
    conv_layer4 = tf.nn.bias_add(tf.nn.conv2d(pooled3, fw4, [1, 1, 1, 1], 'VALID'), b4)
    activated4 = tf.nn.relu(conv_layer4)
    activated4 = tf.nn.dropout(activated4, conv_keep_prob)'''
    
    flattened = tf.concat_v2([flatten(activated3), flatten(pooled2)], 1)
    flattened = tf.nn.dropout(flattened, fc_keep_prob)
    
    fc_w3, fc_b3 = tf.Variable(tf.random_normal((800, n_classes))), tf.zeros((n_classes))
    fc_out3 = tf.add(tf.matmul(flattened, fc_w3), fc_b3)
    return tf.nn.relu(fc_out3)
def tweaked_lenet(x):  # 32 x 32 x 1 
    fw1, b1 = get_conv_filter_bias(x, 5, 5, 16)  # 28 X 28 X 16, 16
    conv_layer1 = tf.nn.bias_add(tf.nn.conv2d(x, fw1, [1, 1, 1, 1], 'VALID'), b1)
    activated1 = tf.nn.relu(conv_layer1)
    activated1 = tf.nn.dropout(activated1, conv_keep_prob)
    pooled1 = tf.nn.max_pool(activated1, [1, 2, 2, 1], [1, 2, 2, 1], 'VALID')  # 14 x 14 x 6
    print(pooled1.get_shape())
    
    fw2, b2 = get_conv_filter_bias(pooled1, 5, 5, 32)  # 10 x 10 x 32, 32
    conv_layer2 = tf.nn.bias_add(tf.nn.conv2d(pooled1, fw2, [1, 1, 1, 1], 'VALID'), b2)
    activated2 = tf.nn.relu(conv_layer2)
    activated2 = tf.nn.dropout(activated2, conv_keep_prob)
    pooled2 = tf.nn.max_pool(activated2, [1, 2, 2, 1], [1, 2, 2, 1], 'VALID')  # 5 x 5 x 32
    print(pooled2.get_shape())
    
    fw3, b3 = get_conv_filter_bias(pooled2, 5, 5, 430)  # 1 x 1 x 430, 430
    conv_layer3 = tf.nn.bias_add(tf.nn.conv2d(pooled2, fw3, [1, 1, 1, 1], 'VALID'), b3)
    activated3 = tf.nn.relu(conv_layer3)
    activated3 = tf.nn.dropout(activated3, conv_keep_prob)
    print(activated3.get_shape())

    flattened = flatten(activated3)
    flattened = tf.nn.dropout(flattened, fc_keep_prob)
    print(flattened.get_shape())
    fc_w3, fc_b3 = tf.Variable(tf.random_normal((430, n_classes))), tf.zeros((n_classes))
    fc_out3 = tf.add(tf.matmul(flattened, fc_w3), fc_b3)
    return tf.nn.relu(fc_out3)
def LeNet(x):    
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1
    
    # SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.
    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 1, 6), mean = mu, stddev = sigma))
    conv1_b = tf.Variable(tf.zeros(6))
    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b

    # SOLUTION: Activation.
    conv1 = tf.nn.relu(conv1)

    # SOLUTION: Pooling. Input = 28x28x6. Output = 14x14x6.
    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # SOLUTION: Layer 2: Convolutional. Output = 10x10x16.
    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))
    conv2_b = tf.Variable(tf.zeros(16))
    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
    
    # SOLUTION: Activation.
    conv2 = tf.nn.relu(conv2)

    # SOLUTION: Pooling. Input = 10x10x16. Output = 5x5x16.
    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')

    # SOLUTION: Flatten. Input = 5x5x16. Output = 400.
    fc0   = flatten(conv2)
    
    # SOLUTION: Layer 3: Fully Connected. Input = 400. Output = 120.
    fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))
    fc1_b = tf.Variable(tf.zeros(120))
    fc1   = tf.matmul(fc0, fc1_W) + fc1_b
    
    # SOLUTION: Activation.
    fc1    = tf.nn.relu(fc1)
    fc1 = tf.nn.dropout(fc1, fc_keep_prob)

    # SOLUTION: Layer 4: Fully Connected. Input = 120. Output = 84.
    fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))
    fc2_b  = tf.Variable(tf.zeros(84))
    fc2    = tf.matmul(fc1, fc2_W) + fc2_b
    
    # SOLUTION: Activation.
    fc2    = tf.nn.relu(fc2)
    fc2 = tf.nn.dropout(fc2, fc_keep_prob)

    # SOLUTION: Layer 5: Fully Connected. Input = 84. Output = 10.
    fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 43), mean = mu, stddev = sigma))
    fc3_b  = tf.Variable(tf.zeros(43))
    logits = tf.matmul(fc2, fc3_W) + fc3_b
    
    return logits
def tweaked_lenet2(x):  # 32 x 32 x 1 
    fw1, b1 = get_conv_filter_bias(x, 5, 5, 32)  # 28 X 28 X 32, 32
    conv_layer1 = tf.nn.bias_add(tf.nn.conv2d(x, fw1, [1, 1, 1, 1], 'VALID'), b1)
    activated1 = tf.nn.relu(conv_layer1)
    #activated1 = tf.nn.dropout(activated1, conv_keep_prob)
    pooled1 = tf.nn.max_pool(activated1, [1, 2, 2, 1], [1, 2, 2, 1], 'VALID')  # 14 x 14 x 32
    #pooled1 = tf.nn.dropout(pooled1, conv_keep_prob)
    
    fw2, b2 = get_conv_filter_bias(pooled1, 3, 3, 64)  # 12 x 12 x 64, 64
    conv_layer2 = tf.nn.bias_add(tf.nn.conv2d(pooled1, fw2, [1, 1, 1, 1], 'VALID'), b2)
    activated2 = tf.nn.relu(conv_layer2)
    #activated2 = tf.nn.dropout(activated2, conv_keep_prob)
    pooled2 = tf.nn.max_pool(activated2, [1, 2, 2, 1], [1, 2, 2, 1], 'VALID')  # 6 x 6 x 64
    #pooled2 = tf.nn.dropout(pooled2, conv_keep_prob)
    
    fw3, b3 = get_conv_filter_bias(pooled2, 3, 3, 64)  # 4 x 4 x 64, 64
    conv_layer3 = tf.nn.bias_add(tf.nn.conv2d(pooled2, fw3, [1, 1, 1, 1], 'VALID'), b3)
    activated3 = tf.nn.relu(conv_layer3)
    activated3 = tf.nn.dropout(activated3, conv_keep_prob)
    
    flattened = flatten(activated3)
    flattened = tf.nn.dropout(flattened, fc_keep_prob)
    
    fc_w3, fc_b3 = tf.Variable(tf.random_normal((1024, n_classes))), tf.zeros((n_classes))
    fc_out3 = tf.add(tf.matmul(flattened, fc_w3), fc_b3)
    out = tf.nn.relu(fc_out3)
    return tf.nn.dropout(out, fc_keep_prob)
    #return out
# Hyperparameters
EPOCHS = 65
BATCH_SIZE = 128
learning_rate = 0.0007
# Placeholders for inputs to the architecture
tf.reset_default_graph()
x = tf.placeholder(tf.float32, (None, 32, 32, 1))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, n_classes)
conv_keep_prob = tf.placeholder(tf.float32)
fc_keep_prob = tf.placeholder(tf.float32)
# Define pipeline
logits = tweaked_lenet(x)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
training_operation = optimizer.minimize(loss_operation)
# Model Evaluation

correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))  # Validate in the horizontal direction
accuracy_prediction = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


def evaluate(X_data, y_data):
    num_examples = len(X_data)
    total_accuracy = 0
    sess = tf.get_default_session()
    for offset in range(0, num_examples, BATCH_SIZE):
        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
        accuracy = sess.run(accuracy_prediction, feed_dict={x: batch_x, y: batch_y, conv_keep_prob: 1.0, fc_keep_prob: 1.0})
        total_accuracy += (accuracy * len(batch_x))
    return total_accuracy / num_examples<jupyter_output><empty_output><jupyter_text>Train and Validate the Model<jupyter_code># Saver

saver = tf.train.Saver()<jupyter_output><empty_output><jupyter_text>A validation set can be used to assess how well the model is performing. A low accuracy on the training and validationsets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.<jupyter_code># Training
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)
    
    print("Training...-----")
    print()
    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y, conv_keep_prob: 0.7, fc_keep_prob: 0.5})
        
        training_accuracy = evaluate(X_train, y_train)
        validation_accuracy = evaluate(X_valid, y_valid)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print("Training Accuracy = {:.3f}".format(training_accuracy))
        print()
        
    saver.save(sess, './lenet')
    print("Model saved")<jupyter_output>Training...-----

EPOCH 1 ...
Validation Accuracy = 0.102
Training Accuracy = 0.125

EPOCH 2 ...
Validation Accuracy = 0.574
Training Accuracy = 0.658

EPOCH 3 ...
Validation Accuracy = 0.724
Training Accuracy = 0.809

EPOCH 4 ...
Validation Accuracy = 0.794
Training Accuracy = 0.880

EPOCH 5 ...
Validation Accuracy = 0.822
Training Accuracy = 0.905

EPOCH 6 ...
Validation Accuracy = 0.848
Training Accuracy = 0.930

EPOCH 7 ...
Validation Accuracy = 0.860
Training Accuracy = 0.943

EPOCH 8 ...
Validation Accuracy = 0.873
Training Accuracy = 0.952

EPOCH 9 ...
Validation Accuracy = 0.880
Training Accuracy = 0.962

EPOCH 10 ...
Validation Accuracy = 0.888
Training Accuracy = 0.969

EPOCH 11 ...
Validation Accuracy = 0.893
Training Accuracy = 0.974

EPOCH 12 ...
Validation Accuracy = 0.892
Training Accuracy = 0.976

EPOCH 13 ...
Validation Accuracy = 0.901
Training Accuracy = 0.980

EPOCH 14 ...
Validation Accuracy = 0.892
Training Accuracy = 0.982

EPOCH 15 ...
Validation Accuracy = 0.90[...]<jupyter_text>Test the model (Moment of truth)<jupyter_code>with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver2 = tf.train.import_meta_graph('./lenet.meta')
    saver2.restore(sess, "./lenet")
    test_accuracy = evaluate(X_test, y_test)
    print("Test Set Accuracy = {:.3f}".format(test_accuracy))<jupyter_output>Test Set Accuracy = 0.962
Tensor("Relu_3:0", shape=(?, 43), dtype=float32)<jupyter_text>--- Step 3: Test a Model on New ImagesTo give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.You may find `signnames.csv` useful as it contains mappings from the class id (integer) to the actual sign name.  Load and Output the Images   Loading imagesThe images have been downloaded from the Test section of official dataset website http://benchmark.ini.rub.de/?section=gtsrb&subsection=datasetDownloads.Amonsgt those, the first 6 images have been selected, which have the names like 00000.ppm to 00005.ppmEach image is of shape 52 x 54. Need to reshape it to 32 x 32<jupyter_code>#Loading ...
X_new_images = [cv2.imread('0000%s.ppm' % (i)) for i in range(6)]  # Each image is of shape 52 x 54
X_new_images = [cv2.resize(img, (32, 32)) for img in X_new_images]<jupyter_output><empty_output><jupyter_text>Plotting<jupyter_code>for idx, img in enumerate(X_new_images):
    plt.subplot(231 + idx)
    plt.imshow(img)
plt.show()<jupyter_output><empty_output><jupyter_text>Manually deciding the labels<jupyter_code>y_new_images = np.array([7, 3, 9, 35, 3, 3])<jupyter_output><empty_output><jupyter_text>Predict the Sign Type for Each Image<jupyter_code>X_grayscale_new_images = grayscale(X_new_images)
for idx, img in enumerate(X_grayscale_new_images):
    plt.subplot(231 + idx)
    plt.imshow(img, cmap='gray')
plt.show()
X_new_images = normalize(X_grayscale_new_images)
print(X_new_images.shape)<jupyter_output>(6, 32, 32, 1)<jupyter_text>Analyze Performance<jupyter_code>### Calculate the accuracy for these 5 new images. 
### For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate on these new images.
soft = tf.nn.softmax(logits)
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver3 = tf.train.import_meta_graph('./lenet.meta')
    saver3.restore(sess, "./lenet")
    accuracy_new_images = evaluate(X_new_images, y_new_images)
    print("New Images Accuracy = {:.3f}".format(accuracy_new_images))
    print(sess.run(soft, feed_dict={x: X_new_images, y: y_new_images, fc_keep_prob: 1.0, conv_keep_prob: 1.0}))<jupyter_output>New Images Accuracy = 1.000
[[  6.15510891e-15   1.49265947e-11   8.50427159e-11   1.11872610e-12
    6.45747236e-13   6.54386270e-07   6.15510891e-15   9.99998808e-01
    5.04413379e-07   6.15510891e-15   3.03317232e-10   6.15510891e-15
    1.46630975e-14   6.15510891e-15   8.01972489e-15   6.15510891e-15
    6.15510891e-15   6.15510891e-15   6.15510891e-15   6.15510891e-15
    6.15510891e-15   6.15510891e-15   6.15510891e-15   6.15510891e-15
    6.15510891e-15   6.15510891e-15   6.15510891e-15   6.15510891e-15
    6.15510891e-15   6.15510891e-15   6.15510891e-15   6.15510891e-15
    6.15510891e-15   6.15510891e-15   6.15510891e-15   6.15510891e-15
    6.15510891e-15   6.15510891e-15   6.15510891e-15   6.15510891e-15
    3.47274348e-10   6.15510891e-15   6.15510891e-15]
 [  2.69559469e-15   1.76226140e-13   9.66763336e-11   9.99999881e-01
    2.30690403e-16   1.17270751e-07   2.30690403e-16   2.30690403e-16
    2.30690403e-16   2.30690403e-16   5.83364330e-14   2.30690403e-16
    2.30[...]<jupyter_text>Output Top 5 Softmax Probabilities For Each Image Found on the Web For each of the new images, print out the model's softmax probabilities to show the **certainty** of the model's predictions (limit the output to the top 5 probabilities for each image). [`tf.nn.top_k`](https://www.tensorflow.org/versions/r0.12/api_docs/python/nn.htmltop_k) could prove helpful here. The example below demonstrates how tf.nn.top_k can be used to find the top k predictions for each image.`tf.nn.top_k` will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.Take this numpy array as an example. The values in the array represent predictions. The array contains softmax probabilities for five candidate images with six possible classes. `tf.nn.top_k` is used to choose the three classes with the highest probability:``` (5, 6) arraya = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,         0.12789202],       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,         0.15899337],       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,         0.23892179],       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,         0.16505091],       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,         0.09155967]])```Running it through `sess.run(tf.nn.top_k(tf.constant(a), k=3))` produces:```TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],       [ 0.28086119,  0.27569815,  0.18063401],       [ 0.26076848,  0.23892179,  0.23664738],       [ 0.29198961,  0.26234032,  0.16505091],       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],       [0, 1, 4],       [0, 5, 1],       [1, 3, 5],       [1, 4, 3]], dtype=int32))```Looking just at the first row we get `[ 0.34763842,  0.24879643,  0.12789202]`, you can confirm these are the 3 largest probabilities in `a`. You'll also notice `[3, 0, 5]` are the corresponding indices.  Investigate the probablity distribution for each class in the new image dataset<jupyter_code>softmaxed_logits = tf.nn.softmax(logits)
top_5_probs = tf.nn.top_k(softmaxed_logits, k=5)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver = tf.train.import_meta_graph('./lenet.meta')
    saver.restore(sess, "./lenet")
    top5 = sess.run(top_5_probs, feed_dict={x: X_new_images, y: y_new_images, fc_keep_prob: 1.0, conv_keep_prob: 1.0})
    print(top5)
    predicted_cls = [inst[0] for inst in top5.indices] # printing out the first index of each predicted instance's class
    print('*' * 10 + ' prediction classes based on 1st index are:', predicted_cls, '*' * 10)
    print(predicted_cls == y_new_images)<jupyter_output>TopKV2(values=array([[  9.99998808e-01,   6.54386270e-07,   5.04413379e-07,
          3.47274348e-10,   3.03317232e-10],
       [  9.99999881e-01,   1.17270751e-07,   9.66763336e-11,
          1.76226140e-13,   5.83364330e-14],
       [  1.00000000e+00,   3.25264608e-25,   5.81864242e-29,
          1.98129291e-31,   6.01217937e-33],
       [  9.99307394e-01,   1.43471334e-04,   9.04302506e-05,
          6.56468328e-05,   6.17633632e-05],
       [  9.96157110e-01,   3.84182041e-03,   9.00264354e-07,
          1.05955280e-07,   1.22065185e-08],
       [  1.00000000e+00,   8.22258883e-09,   2.58439781e-13,
          2.01606477e-18,   3.61510110e-19]], dtype=float32), indices=array([[ 7,  5,  8, 40, 10],
       [ 3,  5,  2,  1, 10],
       [ 9, 10, 16, 41,  3],
       [35, 34, 36,  9, 28],
       [ 3,  5,  2, 10,  1],
       [ 3,  5,  2, 10,  1]], dtype=int32))
********** prediction classes based on 1st index are: [7, 3, 9, 35, 3, 3] **********
[ True  True  True  True  True  True]
{'fraction_non_alphanumeric': 0.12569427674474765, 'fraction_numerical': 0.0839773001690413, 'mean_word_length': 4.084254143646409, 'pattern_counts': {'":': 0, '<': 86, '<?xml version=': 0, '>': 90, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '65363', 'license': 'MIT', 'n_tokens_mistral': 13848, 'n_tokens_neox': 11881, 'n_words': 3376, 'path': 'Traffic_Sign_Classifier.ipynb', 'repo_name': 'srikantviswanath/Traffic-Sign-Classifier-', 'size': '33128'}