<jupyter_start><jupyter_text>Vanishing & Exploding Gradients* gradients get smaller as algorithm progresses to lower layers. Eventually GD leaves lower weights virtually unchanged. so training never converges.* gradients can also grow out of control (often seen in RNNs).* [Significant paper](http://goo.gl/1rhAef) - using combo of logistic sigmoid activiation with random weight initialization (normal, mean=0, stdev=1) -- output variance was >> input variance.* logistic activation: function saturates at 0 or 1 with derivative very close to 0 ==> so backpropagation has no gradient to use.![sigmoid](pics/sigmoid.png)  Xavier & He Initialization* For signals to flow properly in both directions, each layer's output variance should equal its input variance.* Recommends initializing connection weights with random settings using ins, outs![init parameters](pics/init-params-for-activation-funcs.png)* Default: *fully_connected()* function uses Xavier initialization w/ uniform distribution. Change to He initialization by using *variance_scaling_initializer()* function<jupyter_code>import tensorflow as tf
from tensorflow.contrib.layers import fully_connected

n_inputs = 28*28
n_hidden1 = 300

X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")

he_init = tf.contrib.layers.variance_scaling_initializer()
hidden1 = fully_connected(X, n_hidden1, weights_initializer=he_init, scope="h1")<jupyter_output><empty_output><jupyter_text>Non-Saturating Activation Functions* ReLU activations suffer from *dying ReLU* problem (they stop emitting anything other than zero).* Workaround: the **leaku ReLU**. Alpha defines leakage; typical set to 0.01.![leaku ReLU](pics/leaky-relu.png)* Also: **randomized leaky ReLU (RReLU)** (randomized alpha)* Also: **parametric leaky RuLE (PReLU)** (alpha can be modified during backprop)* Also: **exponential linear unit (ELU)**. Allows negative values when z<0; non-zero gradient for z<0 (avoids dying units issue); smooth function everywhere. Uses exponential function, so harder to compute. [paper](http://goo.gl/Sdl2P7)![exponential-relu](pics/exponential-relu.png)<jupyter_code># TF doesn't have leaky ReLU predefined, but easy to build.

def leaky_relu(z, name=None):
    return tf.maximum(0.01 * z, z, name=name)

hidden1 = fully_connected(X, n_hidden1, activation_fn=leaky_relu)<jupyter_output><empty_output><jupyter_text>Batch Normalization* [proposed](https://goo.gl/gA4GSP) to solve vanishing/exploding gradients. * Idea: pror to activation function,    1) zero-center & normalize inputs    2) scale & shift result with 2 new params per layer* Net effect: model learns optimal scale & mean of inputs for each layer* Algorithm:![batch normalization](pics/batch-normalization.png)* Does add computational complexity. Consider plain ELU + He initializaton as well.  Batch Normalization with TF* *batch_normalization()* - centers & normalizes inputs* *batch_norm()* - above, plus finds mean, stdev, scaling, offset params* call directly or include it in *fully_connected()* arguments<jupyter_code># use MNIST dataset again
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/")
#setup

import tensorflow as tf
from tensorflow.contrib.layers import batch_norm, fully_connected

tf.reset_default_graph() 

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 100
n_outputs = 10
learning_rate = 0.01

X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int64, shape=(None), name="y")

def leaky_relu(z, name=None):
  return tf.maximum(0.01 * z, z, name=name)

# is_training: tells batch_norm() whether to use current minibatch's mean & stdev 
# (found during training) or use running avgs (during testing)

with tf.name_scope("dnn"):
    hidden1 = fully_connected(X, n_hidden1, activation_fn=leaky_relu, scope="hidden1")
    hidden2 = fully_connected(hidden1, n_hidden2, activation_fn=leaky_relu, scope="hidden2")
    logits = fully_connected(hidden2, n_outputs, activation_fn=None, scope="outputs")

with tf.name_scope("loss"):
    xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
    loss = tf.reduce_mean(xentropy, name="loss")

with tf.name_scope("train"):
    optimizer = tf.train.GradientDescentOptimizer(learning_rate)
    training_op = optimizer.minimize(loss)

with tf.name_scope("eval"):
    correct = tf.nn.in_top_k(logits, y, 1)
    accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

init = tf.global_variables_initializer()
saver = tf.train.Saver()
n_epochs = 20
batch_size = 100

with tf.Session() as sess:
    
    init.run()
    
    for epoch in range(n_epochs):
        
        for iteration in range(len(mnist.test.labels)//batch_size):
            
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
            
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_test = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})
        
        print(epoch, "Train accuracy:", acc_train, "Test accuracy:", acc_test)

    save_path = saver.save(sess, "my_model_final.ckpt")<jupyter_output>0 Train accuracy: 0.6 Test accuracy: 0.642
1 Train accuracy: 0.73 Test accuracy: 0.7824
2 Train accuracy: 0.81 Test accuracy: 0.827
3 Train accuracy: 0.84 Test accuracy: 0.8539
4 Train accuracy: 0.8 Test accuracy: 0.8686
5 Train accuracy: 0.87 Test accuracy: 0.8759
6 Train accuracy: 0.85 Test accuracy: 0.8843
7 Train accuracy: 0.91 Test accuracy: 0.8903
8 Train accuracy: 0.86 Test accuracy: 0.8969
9 Train accuracy: 0.91 Test accuracy: 0.9018
10 Train accuracy: 0.91 Test accuracy: 0.9014
11 Train accuracy: 0.86 Test accuracy: 0.9065
12 Train accuracy: 0.88 Test accuracy: 0.9078
13 Train accuracy: 0.87 Test accuracy: 0.91
14 Train accuracy: 0.93 Test accuracy: 0.911
15 Train accuracy: 0.9 Test accuracy: 0.9123
16 Train accuracy: 0.91 Test accuracy: 0.9141
17 Train accuracy: 0.9 Test accuracy: 0.9149
18 Train accuracy: 0.92 Test accuracy: 0.9159
19 Train accuracy: 0.93 Test accuracy: 0.9174<jupyter_text>Gradient Clipping* to limit exploding gradients problem. Clip during backprop.* Typical use case: recurrent NNs. * [source](http://goo.gl/dRDAaf)* Uses TF *minimize()* function in optimizer.<jupyter_code>threshold = 1.0

optimizer = tf.train.GradientDescentOptimizer(
    learning_rate)

grads_and_vars = optimizer.compute_gradients(
    loss)

capped_gvs = [
    (tf.clip_by_value(
        grad, -threshold, threshold), var)
    for grad, var in grads_and_vars]

training_op = optimizer.apply_gradients(capped_gvs)<jupyter_output><empty_output><jupyter_text>Pretrained Layers & Reuse* best practice: look for existing NN that tackles similar task, then reuse lower layers (aka *transfer learning*).<jupyter_code># Reuse with TF<jupyter_output><empty_output><jupyter_text>Reusing Models from Other Frameworks* Requires manual loading of weights (ex: Theano)* Very tedious<jupyter_code>'''
original_w = [] # Load the weights from the other framework
original_b = [] # Load the biases from the other framework

X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
hidden1 = fully_connected(X, n_hidden1, scope="hidden1")

[...] # # Build the rest of the model

# Get a handle on the variables created by fully_connected()

with tf.variable_scope("", default_name="", reuse=True): # root scope
    hidden1_weights = tf.get_variable("hidden1/weights")
    hidden1_biases = tf.get_variable("hidden1/biases")

# Create nodes to assign arbitrary values to the weights and biases
original_weights = tf.placeholder(tf.float32, shape=(n_inputs, n_hidden1))
original_biases = tf.placeholder(tf.float32, shape=(n_hidden1))

assign_hidden1_weights = tf.assign(hidden1_weights, original_weights)
assign_hidden1_biases = tf.assign(hidden1_biases, original_biases)

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)
    sess.run(
        assign_hidden1_weights, 
        feed_dict={original_weights: original_w})
        
    sess.run(
        assign_hidden1_biases, 
        feed_dict={original_biases: original_b})
        
    [...] # Train the model on your new task
'''<jupyter_output><empty_output><jupyter_text>Freezing Lower Layers* If 1st DNN already learned low-level features, try to reuse them by freezing the weights.* simplest way:<jupyter_code># provide all trainable var in hidden layers 3,4 & outputs to optimizer function
# (this omits vars in hidden layers 1,2)
'''
train_vars = tf.get_collection(
    tf.GraphKeys.TRAINABLE_VARIABLES,
    scope="hidden[34]|outputs")

# minimizer can't touch layers 1,2 - they're "frozen"

training_op = optimizer.minimize(
    loss, 
    var_list=train_vars)
'''<jupyter_output><empty_output><jupyter_text>Caching Lower Layers* Huge speed boost!<jupyter_code>'''import numpy as np

n_epochs = 100
n_batches = 500

for epoch in range(n_epochs):
    shuffled_idx = rnd.permutation(
        len(hidden2_outputs))
    
    hidden2_batches = np.array_split(
        hidden2_outputs[shuffled_idx], 
        n_batches)
    
y_batches = np.array_split(
    y_train[shuffled_idx], 
    n_batches)

for hidden2_batch, y_batch in zip(hidden2_batches, y_batches):
    sess.run(
        training_op, 
        feed_dict={hidden2: hidden2_batch, y: y_batch})
'''<jupyter_output><empty_output><jupyter_text>Tweaking/Dropping/Replacing Upper Layers* original output layer: should be replaced (little chance of reuse)* iterative freeze/train/compare process to see how many upper layers needed  Model Zoos* When you want to find a net already trained on a similar task* [TensorFlow Model Zoo](https://github.com/tensorflow/models)* [Caffe Model Zoo](https://goo.gl/XI02X3) - converter on [github](https://github.com/ethereon/caffe-tensorflow)  Unsupervised pre-training* Tough problem, but doable.* Train layers one-by-one, starting with lowest layer* Freeze completed layers & train next layer on previous results![unsupervised pretraining](pics/unsupervised-pretraining.png)  Pre-training on easily labeled data - reuse lower layers for "real" task* Often required due to cost/availability of large labeled datasets* Common tactic: label all training data as "good", generate & corrupt additional instances, label new ones as "bad.  Faster Optimizers* Training speedup strategies thus far: 1) smart weight initializations, 2) smart activation functions, 3) batch normalization, 4) reuse of pretraining.* Better optimizer choices:    * Momentum optimization    * Nesterov Accelerated Gradients    * AdaGrad    * RMSProp    * Adam (should almost always use this one)    * Worth noting: below techniques rely on 1st-order partial derivatives (Jacobians); more techniques in literature use 2nd-order derivs (Hessians). Not viable for most deep learning due to memory & computational requirements.  Momentum optimization* local gradient added to a **momentum vector** (m) multiplied by learning rate (n)* ie, **gradient used as an accelerant - not as a speed.*** *beta* hyperparameter serves as friction mechanism. 0 = high friction, 1 = no friction.* Momentum optimization escapes plateaus much faster than GD.<jupyter_code># in TF

optimizer = tf.train.MomentumOptimizer(
    learning_rate=learning_rate,
    momentum=0.9)<jupyter_output><empty_output><jupyter_text>Nesterov Accelerated Gradient* idea: measure cost function gradient *slightly ahead in direction of momentum*.![nesterov vs regular momentum optimization](pics/nesterov.png)<jupyter_code># in TF

optimizer = tf.train.MomentumOptimizer(
    learning_rate=learning_rate,
    momentum=0.9, 
    use_nesterov=True)<jupyter_output><empty_output><jupyter_text>AdaGrad* Scales gradient vector along steepest dimensions, ie it decays the learning rate faster for steep dimensions. (ie *adaptive learning rate*)* Works on simple quadratic problems but often stops too early.![adagrad vs gradient descent](pics/adagrad.png)  RMSProp* Fixes AdaGrad problem by accumulating most recent gradients (instead of all). * Better than AdaGrad on all but very simple problems. Also better than MO and Nesterov.<jupyter_code># in TF

optimizer = tf.train.RMSPropOptimizer(
    learning_rate=learning_rate,
    momentum=0.9, 
    decay=0.9, 
    epsilon=1e-10)<jupyter_output><empty_output><jupyter_text>Adam Optimization ([paper:](https://goo.gl/Un8Axa))* Keeps track of decaying past gradients (like Momentum Optimization)* Keeps track of decaying past squared gradients (like RMSProp)* Default params in TF:* Momentum decay param (beta1) usually set to 0.9* Scaling decay param (beta2) usually set to 0.999* Smoothing term (epsilon) usually set to 10e-8<jupyter_code># in TF

optimizer = tf.train.AdamOptimizer(
    learning_rate=learning_rate)<jupyter_output><empty_output><jupyter_text>Learning Rate Scheduling<jupyter_code># in TF

initial_learning_rate = 0.1
decay_steps = 10000
decay_rate = 1/10

global_step = tf.Variable(
    0, trainable=False)

learning_rate = tf.train.exponential_decay(
    initial_learning_rate, 
    global_step,
    decay_steps, 
    decay_rate)

optimizer = tf.train.MomentumOptimizer(
    learning_rate, 
    momentum=0.9)

training_op = optimizer.minimize(
    loss, 
    global_step=global_step)<jupyter_output><empty_output><jupyter_text>Regularization Techniques  Early Stopping* Simply interrupt training when validation performance starts dropping.  L1 & L2 Regularlization  Dropout* Popular technique - typically adds 1-2% accuracy boost* At every training step, every neuron has probability (p) of being temporarily ignored* Typical p = 50%* In TF: apply *dropout()* to input layer & output of every hidden layer.<jupyter_code># in TF

from tensorflow.contrib.layers import dropout

[...]

is_training = tf.placeholder(
    tf.bool, 
    shape=(), 
    name='is_training')

keep_prob = 0.5

X_drop = dropout(
    X, 
    keep_prob, 
    is_training=is_training)
    
hidden1      = fully_connected(
    X_drop, n_hidden1, scope="hidden1")
    
hidden1_drop = dropout(
    hidden1, keep_prob, is_training=is_training)
    
hidden2      = fully_connected(
    hidden1_drop, n_hidden2, scope="hidden2")
    
hidden2_drop = dropout(
    hidden2, keep_prob, is_training=is_training)
    
logits       = fully_connected(
    hidden2_drop, n_outputs, 
    activation_fn=None,
    scope="outputs")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09660393082852976, 'fraction_numerical': 0.025557330370164593, 'mean_word_length': 4.928365582544257, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 59, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '395658', 'license': 'MIT', 'n_tokens_mistral': 4793, 'n_tokens_neox': 4455, 'n_words': 1372, 'path': 'ch02_basics/ch11-DNN-training.ipynb', 'repo_name': 'pythonProjectLearn/TensorflowLearning', 'size': '14399'}