<jupyter_start><jupyter_text>Transfer LearningTransfer learning: repurposing a network for a different task.<NAME>: wrote library than became CUDNN. Lol.GPU: Optimisod far throughput computation (simult)    - since update lots of pixels on screen at the same time.    - dl computations involve a lot of parallelisation.CPU: latency -> running a single thread of instructions as quickly as possibleRule of thumb: GPU 5x faster than CPU.Stanford University Notes on Transfer Learning: http://cs231n.github.io/transfer-learning/Why transfer learning is useful* Can use intel stored in trained networks to accelerate own progress- Your dataset may be small, so networks pre-trained on much data can help you generalise.DL more popular recently because of- Increased availability of labelled data- Increase in computational power- 1990s LeNet- ImageNet image classification competition     - 2012 AlexNet (Fundamental architecture        - Used parallelism of 40-byte GPUs to train network in about a week.        - Pioneered use of ReLUs in activation.        - Dropout to avoid overfitting.        - 15% error vs 26% error of winner in 2011.-   LabsTransfer Learning with TensorFlowTransfer learning is the practice of starting with a network that has already been trained, and then applying that network to your own problem.Because neural networks can often take days or even weeks to train, transfer learning (i.e. starting with a network that somebody else has already spent a lot of time training) can greatly shorten training time.So how we do go about applying transfer learning? Two popular methods are feature extraction and finetuning.Feature extraction. We take a pretrained neural network and replace the final layer (classification layer) with a new classification layer for the new dataset or perhaps even a small neural network (eventually has a classification layer). During training the weights in all the old layers are frozen, only the weights for the new layer we added are trained. In other words, the gradient doesn't flow backwards past the first new layer we add.Finetuning. Essentially the same as feature extraction except the weights of the old model aren't frozen. The network is trained end-to-end.In this lab and the one later in the section we're going to focus on feature extraction since it's less computationally intensive. Feature ExtractionThe problem is that AlexNet was trained on the ImageNet database, which has 1000 classes of images. You can see the classes in the caffe_classes.py file. None of those classes involves traffic signs.In order to successfully classify our traffic sign images, you need to remove the final, 1000-neuron classification layer and replace it with a new, 43-neuron classification layer.This is called feature extraction, because you're basically extracting the images features captured by the penultimate layer, and passing them to a new classification layer. SolutionFirst, I figure out the shape of the final fully connected layer, in my opinion this is the trickiest part. To do that I have to figure out the size of the output from fc7. Since it's a fully connected layer I know it's shape will be 2D so the second (or last) element of the list will be the size of the output. fc7.get_shape().as_list()[-1] does the trick. I then combine this with the number of classes for the Traffic Sign dataset to get the shape of the final fully connected layer, shape = (fc7.get_shape().as_list()[-1], nb_classes). The rest of the code is just the standard way to define a fully connected in TensorFlow. Finally, I calculate the probabilities via softmax, probs = tf.nn.softmax(logits).<jupyter_code>import time
import tensorflow as tf
import numpy as np
import pandas as pd
from scipy.misc import imread
from alexnet import AlexNet

sign_names = pd.read_csv('signnames.csv')
nb_classes = 43

x = tf.placeholder(tf.float32, (None, 32, 32, 3))
resized = tf.image.resize_images(x, (227, 227))

# Returns the second final layer of the AlexNet model,
# this allows us to redo the last layer for the specifically for 
# traffic signs model.
fc7 = AlexNet(resized, feature_extract=True)
shape = (fc7.get_shape().as_list()[-1], nb_classes)
fc8W = tf.Variable(tf.truncated_normal(shape, stddev=1e-2))
fc8b = tf.Variable(tf.zeros(nb_classes))
logits = tf.nn.xw_plus_b(fc7, fc8W, fc8b)
probs = tf.nn.softmax(logits)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

# Read Images
im1 = imread("construction.jpg").astype(np.float32)
im1 = im1 - np.mean(im1)

im2 = imread("stop.jpg").astype(np.float32)
im2 = im2 - np.mean(im2)

# Run Inference
t = time.time()
output = sess.run(probs, feed_dict={x: [im1, im2]})

# Print Output
for input_im_ind in range(output.shape[0]):
    inds = np.argsort(output)[input_im_ind, :]
    print("Image", input_im_ind)
    for i in range(5):
        print("%s: %.3f" % (sign_names.ix[inds[-1 - i]][1], output[input_im_ind, inds[-1 - i]]))
    print()

print("Time: %.3f seconds" % (time.time() - t))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07070506454816286, 'fraction_numerical': 0.016683217477656404, 'mean_word_length': 4.552370452039692, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '397508', 'license': 'MIT', 'n_tokens_mistral': 1430, 'n_tokens_neox': 1343, 'n_words': 672, 'path': 'examples/deep-learning-notes-and-labs/13_Transfer_Learning/L13_Transfer_Learning.ipynb', 'repo_name': 'kcavagnolo/udacity_selfdrive', 'size': '5044'}