<jupyter_start><jupyter_text>Setup<jupyter_code>import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import time
import itertools as it

import os.path as osp
import glob
import helpers_08

%matplotlib inline<jupyter_output><empty_output><jupyter_text>This notebook incorporates a ton of new concepts.  As such, it is generally more of a walk-through of example code.  There are a few exercises, but we would strongly encourage you to modify the sample code here and experiment with it.  Also, the notebook is written to work on a pure CPU-only system.  There are several points where there are comments in the code with some instructions you can follow to run the code when you have a GPU available.  Have fun!  Variable Scope   `tf.variable_scope()`Throughout the class, we've created our variables directly, using `tf.Variable()`. This is the simplest way to use Variables, as it doesn't involve any programming "magic". However, TensorFlow includes another way to create Variables so that it is easier to access previously created Variables. It also forces you to be more precise with how you use Variables, and allows you to assign "presets" for various parameters in your Variables, such as the initialization values.Let's look at an example:<jupyter_code>var_graph = tf.Graph()
with var_graph.as_default(), tf.variable_scope('my_var_scope'):
    w_init = tf.truncated_normal_initializer()
    b_init = tf.zeros_initializer()
    w = tf.get_variable('w', shape=[10, 10], initializer=w_init)
    b = tf.get_variable('b', shape=[10], initializer=b_init)

print(w,b,sep="\n")<jupyter_output><tf.Variable 'my_var_scope/w:0' shape=(10, 10) dtype=float32_ref>
<tf.Variable 'my_var_scope/b:0' shape=(10,) dtype=float32_ref><jupyter_text>The above code creates two variables, `w` and `b`, using the [`tf.get_variable()`](https://www.tensorflow.org/api_docs/python/tf/get_variable) method. The primary parameter is the string `name` of the `Variable` you'd like to retrieve. If a `Variable` in the scope already has that name, it will retrieve that `Variable` object. Otherwise, it will create that `Variable`. Because neither `w` nor `b` were created before, it creates them from scratch. Reusing variablesNow if we want to reuse them at a later time, we can access them by calling the `variable_scope` again, and setting its `reuse` parameter to `True`:<jupyter_code>with var_graph.as_default(),  tf.variable_scope('my_var_scope', reuse=True):
    w_again = tf.get_variable('w')
    b_again = tf.get_variable('b')

print(w_again, b_again, sep="\n")
print(w is w_again and b is b_again)  # "is" is object identity:  names refer to -same- object<jupyter_output><tf.Variable 'my_var_scope/w:0' shape=(10, 10) dtype=float32_ref>
<tf.Variable 'my_var_scope/b:0' shape=(10,) dtype=float32_ref>
True<jupyter_text>Note that if we are within the same variable scope, we _must_ set `reuse` to `True`. If we don't, TensorFlow will complain at us:<jupyter_code>with var_graph.as_default(), tf.variable_scope('my_var_scope'):
    try:
        w_again = tf.get_variable('w')
        b_again = tf.get_variable('b')
    except ValueError as e:
        print(str(e).splitlines()[0]) # clean it up for quick reading<jupyter_output><empty_output><jupyter_text>As an alternative to passing `reuse` into the `variable_scope` parameter, we can set it after the fact by using the `variable_scope.<jupyter_code>with var_graph.as_default(), tf.variable_scope('my_var_scope') as var_scope:
    var_scope.reuse_variables()
    w_again = tf.get_variable('w')
    b_again = tf.get_variable('b')

print(w_again, b_again, sep="\n")
print(w is w_again and b is b_again)  # "is" is object identity:  names refer to -same- object<jupyter_output><empty_output><jupyter_text>You can get the current variable scope with `tf.get_variable_scope()`; similar to `tf.get_default_graph()`:<jupyter_code>with var_graph.as_default(), tf.variable_scope('my_var_scope'):
    curr_scope = tf.get_variable_scope()
    curr_scope.reuse_variables()
    w_again = tf.get_variable('w')
    b_again = tf.get_variable('b')<jupyter_output><empty_output><jupyter_text>Why would you have to do this?  Imagine that you are a helper function that creates a layer.  If someone outside of you called you to make a layer and it had already defined the `variable_scope`, you might need to get access to that information.  `get_variable_scope` is how you'd do that.  Variable initializersHere's the first example we used above:<jupyter_code>var_graph = tf.Graph()
with var_graph.as_default(), tf.variable_scope('my_var_scope') as scope:
    w_init = tf.truncated_normal_initializer()
    b_init = tf.zeros_initializer()
    w = tf.get_variable('w', shape=[10, 10], initializer=w_init)
    b = tf.get_variable('b', shape=[10], initializer=b_init)

print(w,b,sep="\n")<jupyter_output><empty_output><jupyter_text>Notice that we used two Operations we've never seen before: `truncated_normal_initializer` and `zeros_initializer`. They are similar to what we've used in the past to initialize Variables: `truncated_normal` and `zeros`. The `initializer` Operations are designed to be used with `tf.get_variable()`.  They define a way to create an arbitrary initial value inside a Tensor, regardless of shape.Notice how we don't specify the shape of the `Variable` until we call `tf.get_variable()`. This separation allows us to reuse the same initialization `Operation` for multiple Variables:<jupyter_code>var_graph = tf.Graph()
with var_graph.as_default(), tf.variable_scope('my_var_scope') as scope:
    # common initializer
    w_init = tf.truncated_normal_initializer()

    w1 = tf.get_variable('w1', shape=[10, 10], initializer=w_init)
    w2 = tf.get_variable('w2', shape=[200], initializer=w_init)
    w3 = tf.get_variable('w3', shape=[300,10,10], initializer=w_init)<jupyter_output><empty_output><jupyter_text>Setting default parametersAbove, we use the line `initializer=w_init` over and over again. It would be nice if we could have that automatically be done for us. Luckily, we can! The `variable_scope()` function includes several options that we can provide as a default for any `Variables` we create inside that scope. To set `w_init` as our default initializer, we simple pass in `initializer=w_init` inside of the call to `variable_scope`. Then, we can leave the `initializer=` portion out of `get_variable()`<jupyter_code>var_graph = tf.Graph()
with var_graph.as_default():
    w_init = tf.truncated_normal_initializer()
    with tf.variable_scope('my_var_scope', initializer=w_init) as var_scope:
        w1 = tf.get_variable('w1', shape=[10, 10])
        w2 = tf.get_variable('w2', shape=[200])
        w3 = tf.get_variable('w3', shape=[300,10,10])<jupyter_output><empty_output><jupyter_text>These default parameters can be nested, too.<jupyter_code>var_graph = tf.Graph()
with var_graph.as_default():
    init1 = tf.truncated_normal_initializer()
    init2 = tf.zeros_initializer()
    with tf.variable_scope('var_scope_1', initializer=init1) as var_scope_1:
        w1 = tf.get_variable('w1', shape=[10, 10])
        with tf.variable_scope('var_scope_2', initializer=init2) as var_scope_2:
            w2 = tf.get_variable('w2', shape=[200])
            w3 = tf.get_variable('w3', shape=[300,10,10])<jupyter_output><empty_output><jupyter_text>Configuration Proto When creating a `Session`, you can pass in a set of options in the form of a ConfigProto protocol buffer. You simply add the options you want to the `ConfigProto` initialization function. Here are a couple of the things you can do with the config proto:* Quietly place Ops on different devices if you explicitly call `with tf.device()` using a device that doesn't exist* Print where devices are placed as they are created* Tell TensorFlow to automatically use all of GPU memory immediately, but rather allocate it as necessary* Set a timeout for OperationsWe're going to make use of `log_device_placement` to keep write out (in the terminal) the placement of ops on devices.  And, in `allow_soft_placement` let's TensorFlow *try* to use your `device()` directives, but it can fall back if they aren't possible (bad hardware or an op doesn't exist for a device).<jupyter_code># simple config
sample_config=tf.ConfigProto(allow_soft_placement=True,
                             log_device_placement=True)   
# multi-part config
gpu_config = tf.GPUOptions(allow_growth=True)
multi_config=tf.ConfigProto(allow_soft_placement=True,
                            log_device_placement=True,
                            gpu_options=gpu_config)
sess = tf.Session(config=multi_config)<jupyter_output><empty_output><jupyter_text>Multiple Device TensorFlow By default in TensorFlow, Operations are automatically placed on a CPU or GPU (if available). In general, an Operation will be automatically placed on a GPU unless there isn't a GPU implementation of that `Operation` (assuming you have TensorFlow installed for GPUs).There can be a number of GOTCHAs when dealing with GPU setup.  For a nice collection of configuration tweaks, see:  * https://stackoverflow.com/documentation/tensorflow/10621/tensorflow-gpu-setupt=201707111502593830241In particular, you may need to fiddle with `CUDA_VISIBLE_DEVICES`:  * http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.htmlenv-vars What devices do I have available (and what are their names)?<jupyter_code># list_local_devices is undocumented and may change/disappear
def get_available_device_names():
    from tensorflow.python.client import device_lib
    local_devices = device_lib.list_local_devices()
    return [x.name for x in local_devices]
get_available_device_names()<jupyter_output><empty_output><jupyter_text>Place on Device and Log<jupyter_code>simple_graph = tf.Graph()
with simple_graph.as_default():
    with tf.device('/cpu:0'):
        a = tf.constant(42)
        b = tf.constant(10)

    with tf.device('/cpu:0'): # adjust to /gpu:0 when you have a GPU capable system
        c = tf.multiply(a,b)
# note, the logging will display in the terminal window where you started up jupyter
simple_config=tf.ConfigProto(log_device_placement=True)   

with tf.Session(graph=simple_graph, config=simple_config) as sess:
    print(sess.run(c))<jupyter_output><empty_output><jupyter_text>And here's a demonstration of logging the Op placement on devices.  Unfortunately, we can't capture the output directly in a Jupyter notebook (not even with `%%capture` for you Jupyter super users).  So, you'll need to go to the terminal window where you started jupyter to see the logging messages.  But, they'll look like (we've cleaned up and added some whitespace to make it more readable):    2017-07-11 10:08:16.511206: I tensorflow/core/common_runtime/direct_session.cc:257]     Device mapping:    Mul: (Mul): /job:localhost/replica:0/task:0/cpu:0        2017-07-11 10:08:16.513781: I tensorflow/core/common_runtime/simple_placer.cc:841]         Mul:(Mul)/job:localhost/replica:0/task:0/cpu:0    Const_1: (Const): /job:localhost/replica:0/task:0/cpu:0        2017-07-11 10:08:16.513809: I tensorflow/core/common_runtime/simple_placer.cc:841]         Const_1:(Const)/job:localhost/replica:0/task:0/cpu:0    Const: (Const): /job:localhost/replica:0/task:0/cpu:0        2017-07-11 10:08:16.513817: I tensorflow/core/common_runtime/simple_placer.cc:841]         Const:(Const)/job:localhost/replica:0/task:0/cpu:0  A Small-ish MultiDevice Example  Original Linear RegressionTo get us started, let's look at our old friend, linear regression solve by gradient descent.  To remind us what was going on there, here's some slightly streamlined code:<jupyter_code>class TF_GD_LinearRegression:
    def __init__(s):  
        graph = tf.Graph()
        with graph.as_default():
            with tf.name_scope('inputs'):
                s.x_placeholder = tf.placeholder(tf.float32, [None, 2], name='x')
                s.y_placeholder = tf.placeholder(tf.float32, [None], name='y')
                s.learning_rate = tf.placeholder(tf.float32, [], name='learning_rate')

            with tf.name_scope('model'):
                s.w = tf.Variable(tf.truncated_normal([2, 1]), name='w')
                s.b = tf.Variable(0.0, name='b')
                s.y_hat = tf.matmul(s.x_placeholder, s.w) + s.b            
                s.loss = tf.reduce_mean(tf.square(s.y_hat - s.y_placeholder), name='MSE')
                s.train = tf.train.GradientDescentOptimizer(s.learning_rate).minimize(s.loss)
            
            s.init = tf.global_variables_initializer()
        s.session = tf.Session(graph=graph)
        s.session.run(s.init)
    
    def fit(s, train_dict):
        return s.session.run([s.loss, s.w, s.b, s.train], feed_dict=train_dict)

    def predict(s, test_dict):
        return s.session.run(s.y_hat, feed_dict=test_dict)<jupyter_output><empty_output><jupyter_text>Multi-Device Linear RegressionSo, for using multiple devices in TensorFlow, we often make use of *data parallel* approch where we will duplicate the same model over several devices, do computations on those devices, merge the results, and then iterate.  In the following example, we'll duplicate the linear regression gradient calculations on individual devices and then (1) merge those results (average them) and (2) apply the merged gradients on a central device (typically the CPU in a single host environment).Here, we simulate the multiple devices as all being on the CPU.  However, you can adjust this to use (different) GPUs if you have that hardware.  In the TF docs, replicas of a model are often called "towers".  Our strategy is to compute the gradients on the towers (i.e., we go from training data to gradients).  Our next example is only going to run as far as computing the tower gradients and we're going to just look at the gradients coming off the first tower.  Trust us - it's complicated enough for one step!<jupyter_code>class Multi_LR_V1:
    def __init__(s):  
        graph = tf.Graph()
        with graph.as_default():
            with tf.name_scope('inputs'):
                s.x_placeholder = tf.placeholder(tf.float32, [None, 2], name='x')
                s.y_placeholder = tf.placeholder(tf.float32, [None], name='y')
                s.learning_rate = tf.placeholder(tf.float32, [], name='learning_rate')

            # this effectively "declares" the parameters in a variable scope and explicitly
            # places them on the cpu
            with tf.variable_scope('parameters'), tf.device('/cpu:0'):
                s.w = tf.get_variable('w', [2,1], initializer=tf.truncated_normal_initializer())
                s.b = tf.get_variable('b', [1],   initializer=tf.zeros_initializer())
            with tf.name_scope('train'):
                s.opt = tf.train.GradientDescentOptimizer(s.learning_rate)
                
                # see _build_tower below; pass device= to change target device
                # and/or change default device below
                s.tower_grads = [s._build_tower('tower1'),
                                 s._build_tower('tower2'),
                                 s._build_tower('tower3')]
                # we'll come back to these in a few minutes:
                # s.avg_gradients = process_gradient_list(s.tower_grads)
                # s.train = s.opt.apply_gradients(s.avg_gradients)
            
            
            s.init = tf.global_variables_initializer()
        
        # we could also pass a config into __init__
        simple_config=tf.ConfigProto(log_device_placement=True)   
        s.session = tf.Session(graph=graph)
        s.session.run(s.init)

    # NOTE:  change device parameter to device='/gpu:0' if on GPU capable machine
    def _build_tower(s, name, device='/cpu:0'):
        with tf.name_scope(name), tf.device(device):
            # note, having a class somewhat reduces the need for get_variable/variable_scope
            # but, we're doing this to show how the bigger examples (many of which are not
            # object oriented work)
            with tf.variable_scope('parameters', reuse=True):
                w, b = tf.get_variable('w'), tf.get_variable('b')
            y_hat       = tf.matmul(s.x_placeholder, w) + b
            tower_loss  = tf.square(y_hat - s.y_placeholder)
            tower_grads = s.opt.compute_gradients(tower_loss)
            return tower_grads

        
    def get_one_gradient(s, train_dict):
        return s.session.run(s.tower_grads[0], feed_dict=train_dict)<jupyter_output><empty_output><jupyter_text>Toy Data<jupyter_code>NUM_EXAMPLES = 10
x = np.random.normal(loc=1, scale=1, size=[NUM_EXAMPLES,2])
epsilon = np.random.normal(loc=0, scale=1.5, size=[NUM_EXAMPLES])
y = (3 * x[:,1]) + (2 * x[:,0]) + epsilon<jupyter_output><empty_output><jupyter_text>Gradients From Tower 1 (only)<jupyter_code>model = Multi_LR_V1()

train_dict = {model.x_placeholder: x,
              model.y_placeholder: y,
              model.learning_rate: 0.001}

grad_t1 = model.get_one_gradient(train_dict)
# compute_gradients returns a list of tuples like
# [(gradient 1, variable 1), (gradient 2, variable 2), etc.]

# for the first entry:
grad_t1[0]
# the prior cell output is hard to interpret, 
# until we realize that tensorflow has "numpy-ified" the variable
# but we can see that the first variable there is the current value of "w"
# which is the second element in the tuple in the Out[] immediately above
model.w.eval(session=model.session)
# and then for b 
# (note that our initializer for b was zero and 
#  the second tuple element here is 0)
grad_t1[1]<jupyter_output><empty_output><jupyter_text>So, we will need to take gradients coming off each tower (and for each variable) and "merge" them together into a single gradient to apply to our model parameters.  All the Gradients are belong to usNow, let's scale this up and deal with all of the gradients (and apply them).We are going to assume that the gradients from each tower should be equally weighted (as in the case when each tower seems the same number of examples).  If the towers saw different numbers of examples, we could compute a weighted average of the gradients.  As it is, we can simply average them.Unfortunately, the value of the gradients can be a bit odd - or even worse `None`.  Yes, instead of zero, sometimes the `compute_gradients` function will give us `(None, Variable)` pairs.  Ugh.  According to a comment at:  https://github.com/tensorflow/tensorflow/issues/783, `None` may mean:  * There is no connection from input to output.  * There is a connection, but it's through a discrete variable with meaningless gradients.  * There is a connection, but it's through an op that doesn't have an implemented gradient.  There is some debate in the bug thread about what cases should actually return `0.0`.  However, for our purposes, we just want something that works.  Here's how we'll handle it:  using the structure of the returned gradients (that we diagnosed above), we're going to pull out the individual values, replace `None`s with zero-filled tensors, and average them in.  Easy to say, but the code is a little annoying:<jupyter_code>def process_gradient_list(gradient_list):
    """
    input:
    gradient_list[0] looks like:  [(grad_1, var_1), (grad_2, var_2), ....]
    listifying these looks like: [[(g1,v1), (g2,v2), ...],    list for tower 1
                                  [(g1,v1), (g2,v2), ...],    list for tower 2
                                  [(g1,v1), (g2,v2), ...]  ]  list for tower 3

    Warning: gradient value (i.e., g1) can be None (ugh): 
    see https://github.com/tensorflow/tensorflow/issues/783
    
    zip(*gradient_list) will "unzip" this to:
    [ [(g1, v1), (g1,v1), (g1,v1)]  list for variable/gradient 1
      [(g2, v2), (g2,v2), (g2,v2)]] list for variable/gradient 2
    output:
    single list of: [(avg_grad_1, var_1), (avg_grad_2, var2), .... ]
    """
    def n_to_z(gv_pair):
        'convert None to appropriately shaped zero tensor'
        grad, var = gv_pair
        if grad is None:
            grad = tf.zeros_like(var)
        return grad
    
    avg_of_all_grads = []
    for grads_for_one_var in zip(*gradient_list):
        curr_variable = grads_for_one_var[0][1] # they all have same variable, grab one
        curr_grads    = tf.stack([n_to_z(gv_pair) for gv_pair in grads_for_one_var]) # pull out grads
        curr_avg      = tf.reduce_mean(curr_grads, axis=0)              # average them
        avg_of_all_grads.append((curr_avg, curr_variable))
    
    return avg_of_all_grads<jupyter_output><empty_output><jupyter_text>With that helper, we can now get our replicated model version of linear regression running:<jupyter_code>class TF_Multi_GD_LinearRegression:
    def __init__(s):  
        graph = tf.Graph()
        with graph.as_default():
            with tf.name_scope('inputs'):
                s.x_placeholder = tf.placeholder(tf.float32, [None, 2], name='x')
                s.y_placeholder = tf.placeholder(tf.float32, [None], name='y')
                s.learning_rate = tf.placeholder(tf.float32, [], name='learning_rate')

            with tf.variable_scope('parameters'), tf.device('/cpu:0'):
                s.w = tf.get_variable('w', [2,1], initializer=tf.truncated_normal_initializer())
                s.b = tf.get_variable('b', [1],   initializer=tf.zeros_initializer())

            with tf.name_scope('train'):
                s.opt = tf.train.GradientDescentOptimizer(s.learning_rate)
                
                s.tower_grads = [s._build_tower('tower1'),
                                 s._build_tower('tower2'),
                                 s._build_tower('tower3')]
                s.avg_gradients = process_gradient_list(s.tower_grads)
                s.train = s.opt.apply_gradients(s.avg_gradients)
            
            
            s.init = tf.global_variables_initializer()
        s.session = tf.Session(graph=graph)
        s.session.run(s.init)

    # NOTE:  change device parameter to device='/gpu:0' if on GPU capable machine
    def _build_tower(s, name, device='/cpu:0'):
        with tf.name_scope(name), tf.device(device):
            # note, having a class somewhat reduces the need for get_variable/variable_scope
            # but, we're doing this to show how the bigger examples (many of which are not
            # object oriented work)
            with tf.variable_scope('parameters', reuse=True):
                w, b = tf.get_variable('w'), tf.get_variable('b')
            y_hat       = tf.matmul(s.x_placeholder, w) + b
            tower_loss  = tf.square(y_hat - s.y_placeholder)
            tower_grads = s.opt.compute_gradients(tower_loss)
            return tower_grads

        
    def get_gradients(s, train_dict):
        return s.session.run([s.tower_grads[0], s.tower_grads[1], s.tower_grads[2]], feed_dict=train_dict)
        
    def fit(s, test_dict):
        return s.session.run([s.train], feed_dict=train_dict)
    
    def predict(s, test_dict):
        return s.session.run(s.y_hat, feed_dict=test_dict)
model = TF_Multi_GD_LinearRegression()

train_dict = {model.x_placeholder: x,
              model.y_placeholder: y,
              model.learning_rate: 0.001}

results = model.get_gradients(train_dict)
results[0]<jupyter_output><empty_output><jupyter_text>Records, Queues, and Coordinators These three topics are tightly related.  There are a few TensorFlow guides to give you some insight to their relationships:  * https://www.tensorflow.org/programmers_guide/reading_data  * https://www.tensorflow.org/programmers_guide/threading_and_queuesProbably the major use case for Queues (and Records to feed Queues and Coordinator to drive Queues), is to keep data flowing efficiently to the computational devices (particularly GPUs).  This avoids (1) us writing custom code to cache and transit data from a file archive to our models, (2) let's the executing code all be written at the C++ level for speed.  Less code *and* more efficient?  Yes please!  TFRecords As per usual, we've got some helper code squirreled away that downloads the Tiny ImageNet dataset, which is a miniaturized and simplified version of the ILSVRC dataset. There are only 200 classes as opposed to 1000, and each of the files has been scaled to 64x64 pixels. We're mainly using it due to the fact that it is a smaller dataset!  We also have some code to extract out forms of the labels that we will need.We're going to move the data from the TIN data set format of "directory of files" to TFRecords.  There are two nice reasons to use TFRecords:  (1) they are a TensorFlow native format, so once you convert data to TFRecrods, you can mix-and-match that data to models without worrying about data conversion and (2) it's particularly easy to feed TFRecords to Queues.  Note, you can also feed other data, e.g., lines from csv files to Queues relatively easily.The general process is to gather multiple `Feature` instances into a `Features` and then into an `Example`.  The `Example` can be written out with a `TFRecordWriter`.<jupyter_code>tin_path = helpers_08.grab_tin()
id_to_synset, synset_to_id = helpers_08.get_tin_id_maps(tin_path)
# Common helper function pattern for creating Features
def _int64_feature(value):
    return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
def _bytes_feature(value):
    return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))

def make_example(img, lbl):
    features = tf.train.Features(feature = {'image':_bytes_feature(img),
                                            'label':_int64_feature(lbl)})
    return tf.train.Example(features=features)
def convert_tin_train_to_tfr(tin_path):
    helpers_08.mkdir(osp.join(tin_path, "tfr"))
    tfr_dest_path = osp.join(tin_path, "tfr", "tin_train.tfr")
    base_src_path  = osp.join(tin_path, 'tiny-imagenet-200', "train")
    
    writer = tf.python_io.TFRecordWriter(tfr_dest_path)

    for synset_dir in glob.glob(osp.join(base_src_path, "n*")):
        int_label = synset_to_id[osp.basename(synset_dir)]
        for filename in glob.glob(osp.join(synset_dir, "images/*.JPEG")):
            with open(filename, 'rb') as f:
                image_bytes = f.read()
            example = make_example(image_bytes, int_label)
            writer.write(example.SerializeToString())
    writer.close()

# and "go"; comment out if you are rerunning the notebook
# to save some time
convert_tin_train_to_tfr(tin_path)   
def convert_tin_validation_to_tfr(tin_path):
    tfr_dest_path = osp.join(tin_path, "tfr", "tin_valid.tfr")
    base_src_path  = osp.join(tin_path, 'tiny-imagenet-200', "val")
    
    label_filename = osp.join(base_src_path, "val_annotations.txt")
    def proc_line(line):
        filename, synset = line.split(None, 2)[:2]
        label = synset_to_id[synset]
        return filename, label
    filename_to_label = dict(proc_line(a_line) for a_line in open(label_filename))
    
    writer = tf.python_io.TFRecordWriter(tfr_dest_path)
    
    for long_filename in glob.glob(osp.join(base_src_path, "images/*.JPEG")):
        int_label = filename_to_label[osp.basename(long_filename)]
        with open(long_filename, 'rb') as f:
            image_bytes = f.read()
        example = make_example(image_bytes, int_label)
        writer.write(example.SerializeToString())
    writer.close()

# and "go"; comment out if you are rerunning the notebook
# to save some time
convert_tin_validation_to_tfr(tin_path)<jupyter_output><empty_output><jupyter_text>Exercise To practice with `TFRecords`, grab the MNIST character image data and convert it from the `*.npy` files to tensorflow Records.  Note, the data is probably in your `week_04/data` directory.  Solution For converting MNIST to records:  * https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/convert_to_records.pyIf you want to see how that gets used by a trainer:  * https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/how_tos/reading_data/fully_connected_reader.py  Queues As we said using Queues will reduce our coding overhead and help us efficiently feed our computing devices.  Here's a quick example of a pipeline that goes from TFRecords through some preprocessing and spits out training batches of images and labels.<jupyter_code># can increase these for GPU systems; see docs for tf.train.shuffle_batch
BATCH_SIZE = 5
NUM_THREADS = 2
CAPACITY = 100 + 3 * BATCH_SIZE

def image_label_queue():
    with tf.name_scope('input_queue'):
        # create the list of filenames [see also: tf.train.match_filenames_once()]
        # and queue them up
        train_path = "data/tin/tfr/tin_train.tfr"
        valid_path = "data/tin/tfr/tin_valid.tfr"
        filenames = [train_path, valid_path]
        filename_queue = tf.train.string_input_producer([train_path])

        reader = tf.TFRecordReader()
        key, example = reader.read(filename_queue)
        # we gave these an image and a label above in make_example()
        features = tf.parse_single_example(example, features={'image': tf.FixedLenFeature([], tf.string),
                                                              'label': tf.FixedLenFeature([], tf.int64)})
        image = features['image']
        label = tf.to_int32(features['label'])

        # PREPROCESS BEFORE ADDING TO QUEUE
        image = tf.image.decode_jpeg(image, channels=3)
        image = tf.expand_dims(image, 0)
        image = tf.image.resize_bilinear(image, [224, 224])
        image = tf.squeeze(image)
        image = tf.to_float(image)
        image = (image - 127.5) / 127.5

        # and queue it with shuffle_batch
        image_batch, label_batch = tf.train.shuffle_batch([image, label], 
                                                          batch_size=BATCH_SIZE,
                                                          num_threads=NUM_THREADS,
                                                          capacity=CAPACITY,
                                                          min_after_dequeue=15)
        return image_batch, label_batch<jupyter_output><empty_output><jupyter_text>Linear Regression Coordinator Now, we want to show off how we *use* the Queues to feed our program.  There is a more complicated example coming with VGGNet below.  But here, we'll return to the humble linear regression code and feed it with a `tf.train.shuffle_batch` that is populated from a simple NumPy array (which gets automatically converted to a Tensor for us - as we saw in week 1).We're going to do one "ugly" thing in this example:  to feed the `shuffle_batch`, we're going to pass in the training data to init.  This isn't pretty, but the Coordinator introduces an issue.  To use the Coordinator, we need to `start_queue_runners` which in turn need their dependencies filled-in (aka, placeholders need to be fed).  But, feeding placeholders (and not doing anything else) is surprisingly trickly, so we're working around that.  There are hints that we could use a `placeholder` followed by a `Variable` to get values to `shuffle_batch`, but we were unable make that work (see "Preloaded Data" here: https://www.tensorflow.org/programmers_guide/reading_data).  Better luck to you!To make use of our Queue (which is returned by `tf.train.shuffle_batch`), we need to drive it.  Here is template code to do that:<jupyter_code>def coord_fit(s, train_dict, steps):
    coord = tf.train.Coordinator()
    with coord.stop_on_exception():
        threads = tf.train.start_queue_runners(sess=s.session, coord=coord)
        for step in range(steps):
            if coord.should_stop(): 
                break
    s.session.run(s.train, feed_dict=train_dict)<jupyter_output><empty_output><jupyter_text>That function becomes our `fit` method and away we go:<jupyter_code>class Coord_LinearRegression:
    def __init__(s, in_x, in_y):  
        graph = tf.Graph()
        with graph.as_default():
            with tf.name_scope('train_inputs'):
                s.x_train, s.y_train = tf.train.shuffle_batch([in_x, in_y], 
                                                               batch_size=15,
                                                               num_threads=2,
                                                               capacity=100,
                                                               min_after_dequeue=15,
                                                               enqueue_many = True,
                                                               allow_smaller_final_batch=True)

                s.learning_rate = tf.placeholder(tf.float32, [], name='learning_rate')

            with tf.name_scope('test_inputs'):
                s.x_test = tf.placeholder(tf.float32, [None, 2], name='x_test')
                s.y_test= tf.placeholder(tf.float32, [None], name='y_test')

                
            with tf.name_scope('train_model'):
                s.w = tf.Variable(tf.truncated_normal([2, 1]), name='w')
                s.b = tf.Variable(0.0, name='b')
                s.y_hat_train = tf.matmul(s.x_train, s.w) + s.b
                s.train_loss = tf.reduce_mean(tf.square(s.y_hat_train - s.y_train), name='MSE')
                s.train = tf.train.GradientDescentOptimizer(s.learning_rate).minimize(s.train_loss)
                
            with tf.name_scope('test_model'):
                s.y_hat_test = tf.matmul(s.x_test, s.w) + s.b

                
            
            s.init = tf.global_variables_initializer()
        s.session = tf.Session(graph=graph)
        s.session.run(s.init)
    
    def fit(s, train_dict, steps):
        coord = tf.train.Coordinator()
        with coord.stop_on_exception():
            threads = tf.train.start_queue_runners(sess=s.session, coord=coord)
            for step in range(steps):
                if coord.should_stop(): 
                    break
                s.session.run(s.train, feed_dict=train_dict)
        coord.request_stop()
        coord.join(threads)
        # s.session.close() # keep it open so we can predict


    def predict(s, test_dict):
        return s.session.run(s.y_hat_test, feed_dict=test_dict)
NUM_TRAIN_EXAMPLES = 5000
def make_y(x):
    epsilon = np.random.normal(loc=0, scale=1.5, size=len(x)).astype(np.float32)
    y = (3 * x[:,0]) + (2 * x[:,1]) + epsilon
    return y

# weirdness
def make_input_data():
    x_train = np.random.normal(loc=1, scale=1, size=[NUM_TRAIN_EXAMPLES,2]).astype(np.float32)
    y_train = make_y(x_train)
    return [x_train, y_train]
model = Coord_LinearRegression(*make_input_data()) # detuple
train_dict = {model.learning_rate: 0.001}
model.fit(train_dict, 20)
NUM_TEST_EXAMPLES = 15
x_test = np.random.normal(loc=1, scale=1, size=[NUM_TEST_EXAMPLES,2]).astype(np.float32)
y_test = make_y(x_test)

model.predict({model.x_test:x_test, model.y_test:y_test})[:5]<jupyter_output><empty_output><jupyter_text>An interesting (and slightly aggravating) side-effect of using a Queue to provide our inputs (instead of `tf.placeholder`s), is that we've lost our ability to naturally feed test data to the model by setting the placeholders.  So, you may have noticed that in `Coord_LinearRegression`, we added a `name_scope` for `test_inputs`.  These *can* be set with `feed_dict` and they allow us to use our "normal" prediction technique.  VGGNet Our end goal today is going to be a multi-device and Coordinated version of VGGNet.  That's a pretty big step from where we're at right now.  So, let's break that down into a (large) subtask.  Creating the VGGNet architecture.You can read about VGGNet in the [VGGNet paper on arXiv.org](https://arxiv.org/abs/1409.1556).  Here is the architecture of VGG.  The convolution layers use a *3x3* windows and max pool layers use a *2x2* window.|19 (Wgt) Layer VGG Network||----------------|| 224 x 224 RGB  || 2 conv-64      || max pool || 2 conv-128 || maxpool|| 4 conv-256 | | max pool || 4 conv-512 | | max pool | | 4 conv-512 || max pool || FC 4096 (w/dropout) || FC 4096 (w/dropout) || FC 1000 || soft max|  Loading TensorBoard Graph for a Pre-built VGG ModelInside of the `prebuilt` folder, there is a TensorBoard graphs exported for VGGNet. You will use these as guidance for creating your own VGGNet. Fire up TensorBoard:```shelltensorboard --logdir=prebuilt```Navigate to `localhost:6006` in your browser. After you click on the "Graphs" link, you'll be able to switch to VGG from the drop-down "Runs" menu.  Older Layer functions In `helpers_08.py`, we have gathered together versions of:   * `conv_layer`,   * `flatten`,   * `fully_connected_xavier_relu_layer`, and   * `pool_layer`.    You can use these to build your implementation of VGGNet.  If you want a quick reminder of their code, in a Jupyter notebook can use `??helpers_08.pool_layer` to get the source code of `pool_layer` (and likewise for the other functions).  Of course, you can always open `helpers_08.py` in a text editor.We've made one set of modifications to `conv_layer` and `fully_connected_xavier_relu_layer`:  we've traded a `tf.name_scope` in each for `tf.variable_scope` so that the same layers in different replicas (towers) of our model will use the same parameters which are stored on the CPU.  We discussed this earlier.<jupyter_code># to see helper_08.pool_layer code, uncomment the bottom line 
# (it will open a pop-up at the bottom of your browser):
# ??helpers_08.pool_layer<jupyter_output><empty_output><jupyter_text>Exercise Use the above layer functions to recreate the VGGNet from the above architecture. Your model function should expect two parameter inputs:* `incoming`: a 4D tensor with dtype `float32` and shape `[batch_size, 224, 224, 3]`* `keep_prob`: A scalar `Tensor` with dtype `float32` representing the keep_probability for dropout<jupyter_code>def vgg_net(incoming, keep_prob):
    ### YOUR CODE HERE
    pass
    ###<jupyter_output><empty_output><jupyter_text>When you think you have something reasonable, you can create the network and save it to disk.  Then, open it up in TensorBoard to check it out.  A MultiGPU and Coordinated VGG Net And now for all the marbles.  Here, we put together towers of models (which use `variable_scope` to share parameters) which we can place on devices as we want, we feed data to the model using Queues (fed from `TFRecord`s and controlled by a `Coordinator`).  But wait!  There's more!  We also hook in a VGGNet as the model.  Graduation time:<jupyter_code># since it's integral to the model, this is the image_label_queue from above
BATCH_SIZE = 5
NUM_THREADS = 2
CAPACITY = 100 + 3 * BATCH_SIZE

def image_label_queue():
    with tf.name_scope('input_queue'):
        # create the list of filenames [see also: tf.train.match_filenames_once()]
        # and queue them up
        train_path = "data/tin/tfr/tin_train.tfr"
        valid_path = "data/tin/tfr/tin_valid.tfr"
        filenames = [train_path, valid_path]
        filename_queue = tf.train.string_input_producer([train_path])

        reader = tf.TFRecordReader()
        key, example = reader.read(filename_queue)
        features = tf.parse_single_example(example, features={'image': tf.FixedLenFeature([], tf.string),
                                                              'label': tf.FixedLenFeature([], tf.int64)})
        image = features['image']
        label = tf.to_int32(features['label'])

        # PREPROCESS BEFORE ADDING TO QUEUE
        image = tf.image.decode_jpeg(image, channels=3)
        image = tf.expand_dims(image, 0)
        image = tf.image.resize_bilinear(image, [224, 224])
        image = tf.squeeze(image)
        image = tf.to_float(image)
        image = (image - 127.5) / 127.5

        image_batch, label_batch = tf.train.shuffle_batch([image, label], 
                                                          batch_size=BATCH_SIZE,
                                                          num_threads=NUM_THREADS,
                                                          capacity=CAPACITY,
                                                          min_after_dequeue=15)
        return image_batch, label_batch
class Multi_VGG_Model:
    def __init__(s, num_towers, config):
        DECAY_STEPS, DECAY_FACTOR = 100000, 0.998

        multi_vgg_graph = tf.Graph()
        with multi_vgg_graph.as_default(), tf.device('/cpu:0'):
            with tf.name_scope('step'):
                s.global_step = tf.Variable(0, trainable=False, dtype=tf.int32, name="global_step")
                s.increment = s.global_step.assign_add(1)

            with tf.name_scope('inputs'):
                s.image_batch, s.label_batch = image_label_queue()
                
            with tf.name_scope('train'):
                lr = tf.train.exponential_decay(0.05, s.global_step, 
                                                DECAY_STEPS, DECAY_FACTOR, 
                                                staircase=True)
                s.opt = tf.train.AdamOptimizer(lr)
                s.tower_grads = [s._build_tower('tower_{:02d}'.format(i), i==0) for i in range(num_towers)]
                s.avg_gradients = process_gradient_list(s.tower_grads)
                s.train = s.opt.apply_gradients(s.avg_gradients)
            
            with tf.name_scope('summaries'):    
                for grad, var in s.avg_gradients:
                    if grad is not None:
                        tf.summary.histogram(var.op.name + '_gradients', grad)
                for var in tf.trainable_variables():
                    tf.summary.histogram(var.op.name, var)
                s.merged_summaries = tf.summary.merge_all()

            s.saver = tf.train.Saver()                                 # for model checkpoints
            s.init = tf.global_variables_initializer()

        s.writer = tf.summary.FileWriter("vgg_status/multi_vgg", graph=multi_vgg_graph) # for model summaries
        s.session = tf.Session(graph=multi_vgg_graph, config=config)
        s.session.run(s.init)
    
    # NOTE:  change device parameter to device='/gpu:0' if on GPU capable machine
    def _build_tower(s, name, first_tower, device='/cpu:0'):
        print(name, ' on ', device)
        with tf.name_scope(name), tf.device(device):
            labels_onehot=tf.one_hot(s.label_batch, 1000) # number of classes

            #
            # this is awkward, but basically, it is our mechanism to 
            # have the first tower -create- the shared variables in in the scopes
            # and have subsequent towers -reuse- those variables
            #
            # it works by getting the current scope (very likely, the outermost 
            # -- unless we are called inside someone else's variable_scope)
            # and setting it to reuse if and only if we are a later tower
            #
            with tf.variable_scope(tf.get_variable_scope()) as curr_scope:
                if not first_tower:
                    curr_scope.reuse_variables()
                logits = vgg_net(s.image_batch)

            smce = tf.nn.softmax_cross_entropy_with_logits
            tower_loss = tf.reduce_mean(smce(labels=labels_onehot, logits=logits))
            tower_grads = s.opt.compute_gradients(tower_loss)
            return tower_grads

    def fit(s, steps, summary_step = None, checkpoint_step = None):
        # default these Nones to "never"
        summary_step = summary_step if summary_step else steps + 1      
        checkpoint_step = checkpoint_step if checkpoint_step else steps + 1

        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(sess=s.session, coord=coord)

        with coord.stop_on_exception():
            for step in range(steps):
                # mod is zero and step > 0
                do_summaries  = not step % summary_step and step
                do_checkpoint = not step % checkpoint_step and step

                if coord.should_stop(): 
                    break
                if do_summaries:
                    s.write_summaries()
                else:
                    s.session.run(s.train)
                if do_checkpoint: 
                    s.save_checkpoint()
        coord.request_stop()
        coord.join(threads)
        s.session.close()
    
    def write_summaries(s):
        _, summaries, gs = s.session.run([s.train, s.merged_summaries, s.global_step])
        s.writer.add_summary(summaries, global_step=gs)
        s.writer.flush()
    
    def save_checkpoint(s):
        s.saver.save(s.session, 'vgg_status/multi_vgg/multi_vgg.ckpt', 
                     s.global_step.eval(session=s.session))
config=tf.ConfigProto(allow_soft_placement=True)
                      #log_device_placement=True) # substantial output b/c big model
multi_vgg = Multi_VGG_Model(4, config=config) 
# this gets 800% utilization on my 8 core machine
# a few steps here is enough to "crush" my 8 core; but you can fly through
# 10 steps on a Tesla GPU (AWS p2.xlarge) (COOLNESS!)
multi_vgg.fit(1)<jupyter_output><empty_output><jupyter_text>Show and Tell As an example of how we can repurpose VGGNet, here is a skeleton of code to run Show and Tell.For a full working example in the TensorFlow code base, see here:  * https://github.com/tensorflow/models/tree/master/im2txtNote, that there is both `git:tensorflow/models` as its own project (with many interesting models in it) *and* the `git:tensorflow/tensorflow` project (the mian code base) and a (somewhat more boring) `tensorflow/models` directory.<jupyter_code>## VOCABULARY SETTINGS:  in/out and embedding sizes
IN_VOCAB, IN_EMBED_SIZE = 10000, 100
OUT_VOCAB, OUT_EMBED_SIZE = 10000, 4096

## RNN SETTINGS
RNN_WIDTH, RNN_DEPTH = 1000, 4
MAX_LEN_INPUTS = MAX_LEN_LABELS = 10

## TRAINING SETTINGS
BATCH_SIZE = 32
NUM_NEG_SAMPLES=50

show_graph = tf.Graph()
with show_graph.as_default():
    inputs = tf.placeholder(tf.float32, [None, 224, 224, 3])
    labels = tf.placeholder(tf.int32, [None, MAX_LEN_LABELS])
    labels_length = tf.placeholder(tf.int32, [None])
    
    # Encoder
    vgg_logits = vgg_net(inputs)
    encoded_image = show_graph.get_tensor_by_name('fc_1/Relu:0')
    
    # Decoder
    output_embedding = tf.Variable(tf.truncated_normal([OUT_VOCAB, OUT_EMBED_SIZE]))
    w = tf.Variable(tf.truncated_normal([RNN_WIDTH, OUT_VOCAB]))
    b = tf.Variable(tf.zeros([OUT_VOCAB]))
    
    cell_base = tf.contrib.rnn.GRUCell
    dec_cells = [cell_base(RNN_WIDTH) for _ in range(RNN_DEPTH)]
    dec_cell = tf.contrib.rnn.MultiRNNCell(dec_cells)
    
    # Initial decoder state is zero, instead of encoder state (not an RNN encoder)
    dec_state = tuple(tf.unstack(tf.zeros([RNN_DEPTH,tf.shape(labels)[0], RNN_WIDTH])))
    # Initial input is the encoded image
    output, dec_state = dec_cell(encoded_image, dec_state)
    tf.get_variable_scope().reuse_variables()
    
    # Don't append the output from the first LSTM
    outputs, logits = [], []
    prev = output
    for i, label in enumerate(tf.unstack(labels, axis=1)):
        logit = tf.matmul(prev, w) + b
        logits.append(logit)
        label_idx = tf.argmax(logit, 1)
        label_emb = tf.nn.embedding_lookup(output_embedding, label_idx)
        prev, dec_state = dec_cell(label_emb, dec_state)
        if i > 0:
            outputs.append(prev)

    lengths_exp = tf.expand_dims(labels_length, 1)
    mask = tf.reshape(tf.tile(tf.range(MAX_LEN_LABELS), [tf.shape(labels)[0]]), 
                      [-1, MAX_LEN_LABELS], 
                      name='mask_reshape')
    mask = tf.to_float(tf.less(mask, lengths_exp))
    
    def s_loss(logits, labels):
        logits = tf.reshape(logits, [-1, RNN_WIDTH], name='logit_s')
        labels = tf.reshape(labels, [-1, 1], name='label_s')
        return tf.nn.sampled_softmax_loss(weights=tf.transpose(w), biases=b,
                                          labels=labels, inputs=logits,
                                          num_sampled=NUM_NEG_SAMPLES, num_classes=OUT_VOCAB)
    test = tf.convert_to_tensor(outputs)
    outputs_tensor = tf.reshape(test, [-1, MAX_LEN_LABELS, RNN_WIDTH], name='logit_reshape')

    labels_float = tf.to_float(labels)
    loss = tf.contrib.seq2seq.sequence_loss(outputs_tensor, labels_float, mask, softmax_loss_function=s_loss)
    
    learning_rate = tf.placeholder(tf.float32, [])
    train = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
    
    global_step = tf.Variable(0, trainable=False, name='global_step')
    inc_step = tf.assign_add(global_step, 1, name='inc_step')
    
    logits_tensor = tf.reshape(tf.convert_to_tensor(logits), [-1, MAX_LEN_LABELS, OUT_VOCAB])
    softmax = tf.nn.softmax(logits_tensor)
    predictions = tf.to_int32(tf.argmax(softmax, 2))
    
    init = tf.global_variables_initializer()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09717172961981638, 'fraction_numerical': 0.014644566311335675, 'mean_word_length': 3.074991630398393, 'pattern_counts': {'":': 0, '<': 111, '<?xml version=': 0, '>': 113, 'https://': 11, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 50, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '649379', 'license': 'MIT', 'n_tokens_mistral': 15430, 'n_tokens_neox': 14641, 'n_words': 4418, 'path': 'tutorials/Intel-TF101-Class8/Week8_ILSVRC_and_Device_Targets_HW.ipynb', 'repo_name': 'Nhat-Minh-Hoang-Tran-BSC2021/Intel-Convolutional_Neural_Networks', 'size': '48687'}