<jupyter_start><jupyter_text>TensorBoard Session Lab: tf.estimatorIn this lab, you will visualize the TF graph for a tf.estimator modelHelpful references:- https://www.youtube.com/watch?reload=9&v=eBbEDRsCmv4<jupyter_code>import tensorflow as tf
import time
import os
import numpy as np<jupyter_output>/usr/lib/python3/dist-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters<jupyter_text>Two functions that will be reused in your custom model function._conv:- Convolution block to include conv, relu, and max pool.- Use tf.variable_scope.- Input should include feature, kernel (convolution filter), variable scope name_dense:- Dense block with relu.  Since last layer is logits, make relu switchable on/off.- Use tf.variable_scope.- Since last layer is logits, make relu switchable on/off.- Input should include feature, in/out sizes, variable scope name, relu(true/false)<jupyter_code># Convolution Block

def _conv(x,kernel,name,log=False):
    with tf.variable_scope(name):
        W = tf.get_variable(initializer=tf.truncated_normal(shape=kernel,stddev=0.01),name='W')
        b = tf.get_variable(initializer=tf.constant(0.0,shape=[kernel[3]]),name='b')
        conv = tf.nn.conv2d(x, W, strides=[1,1,1,1],padding='SAME')
        activation = tf.nn.relu(tf.add(conv,b))
        pool = tf.nn.max_pool(activation,ksize=[1,2,2,1],strides=[1,2,2,1],padding='SAME')
        #if log==True:
            #
            # code here
            #
        return pool

# Dense Block

def _dense(x,size_in,size_out,name,relu=False,log=False):
    with tf.variable_scope(name):
        flat = tf.reshape(x,[-1,size_in])
        W = tf.get_variable(initializer=tf.truncated_normal([size_in,size_out],stddev=0.1),name='W')
        b = tf.get_variable(initializer=tf.constant(0.0,shape=[size_out]),name='b')
        activation = tf.add(tf.matmul(flat,W),b)
        if relu==True:
            activation = tf.nn.relu(activation)
        #if log==True:
            #
            # code here
            #
        return activation<jupyter_output><empty_output><jupyter_text>Custom model function.5 sections of the model function 1. INFERENCE MODELRecommended architecture:- conv1: kernel = 5x5x128- conv2: kernel = 5x5x128- conv3: kernel = 3x3x256- conv4: kernel = 3x3x512- dense: hidden_units (tunable hyperparam)Use:- tf.nn - this will allow for metric collection later. 2. CALCULATIONS AND METRICSImplement:- Prediction dictionary {classes, logits, probabilities}.- Loss function: Cross Entropy.- Accuracy for both training and eval using tf.metrics. 3. MODE = PREDICTImplement:- EstimatorSpec for PREDICT. 4. MODE = TRAINImplement:- Optimizer = Stochastic Gradient Descent.- EstimatorSpec for TRAIN.- Optional: Exponential Decay Learning Rate. 5. MODE = EVALImplement:- EstimatorSpec for EVAL.  1a - Graph with bug<jupyter_code>def cnnmodel_fn(features, labels, mode, params):
    
    #### 1 INFERNCE MODEL

   
    input_layer = tf.reshape(features, [-1, 32, 32, 3])
    conv1 = _conv(input_layer,kernel=[5,5,3,128],name='conv1',log=params['log'])
    conv2 = _conv(conv1,kernel=[5,5,128,128],name='conv2',log=params['log'])
    conv3 = _conv(conv2,kernel=[3,3,128,256],name='conv3',log=params['log'])
    conv4 = _conv(conv2,kernel=[5,5,128,256],name='conv4',log=params['log'])
    dense = _dense(conv2,size_in=2*2*2048,size_out=params['dense_units'],
                   name='Dense',relu=True,log=params['log'])
    if mode==tf.estimator.ModeKeys.TRAIN:
        dense = tf.nn.dropout(dense,params['drop_out'])
    logits = _dense(dense,size_in=params['dense_units'],
                    size_out=10,name='Output',relu=False,log=params['log'])
        
    #### 2 CALCULATIONS AND METRICS
    
    predictions = {"classes": tf.argmax(input=logits,axis=1),
                   "logits": logits,
                   "probabilities": tf.nn.softmax(logits,name='softmax')}
    export_outputs = {'predictions': tf.estimator.export.PredictOutput(predictions)}
    if (mode==tf.estimator.ModeKeys.TRAIN or mode==tf.estimator.ModeKeys.EVAL):
        loss = tf.losses.sparse_softmax_cross_entropy(labels=labels,logits=logits)
        accuracy = tf.metrics.accuracy(
            labels=labels, predictions=tf.argmax(logits,axis=1))
        metrics = {'accuracy':accuracy}
        
    #### 3 MODE = PREDICT
    
    if mode == tf.estimator.ModeKeys.PREDICT:
        return tf.estimator.EstimatorSpec(
            mode=mode, predictions=predictions, export_outputs=export_outputs)

    #### 4 MODE = TRAIN

    if mode == tf.estimator.ModeKeys.TRAIN:
        learning_rate = tf.train.exponential_decay(
            params['learning_rate'],tf.train.get_global_step(),
            decay_steps=100000,decay_rate=0.96)
        optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
        train_op = optimizer.minimize(loss=loss,global_step=tf.train.get_global_step())
        #
        #
        #
        return tf.estimator.EstimatorSpec(
            mode=mode, loss=loss, train_op=train_op)
    
    #### 5 MODE = EVAL
    
    if mode == tf.estimator.ModeKeys.EVAL:
        return tf.estimator.EstimatorSpec(
            mode=mode,loss=loss,eval_metric_ops=metrics)<jupyter_output><empty_output><jupyter_text>STOP DO NOT RUN SECTION 1b 1b - Fixed Graph<jupyter_code>def cnnmodel_fn(features, labels, mode, params):
    
    #### 1 INFERNCE MODEL
    
    input_layer = tf.reshape(features, [-1, 32, 32, 3])
    conv1 = _conv(input_layer,kernel=[5,5,3,128],name='conv1',log=params['log'])
    conv2 = _conv(conv1,kernel=[5,5,128,128],name='conv2',log=params['log'])
    conv3 = _conv(conv2,kernel=[3,3,128,256],name='conv3',log=params['log'])
    conv4 = _conv(conv3,kernel=[3,3,256,512],name='conv4',log=params['log'])
    dense = _dense(conv4,size_in=2*2*512,size_out=params['dense_units'],
                   name='Dense',relu=True,log=params['log'])
    if mode==tf.estimator.ModeKeys.TRAIN:
        dense = tf.nn.dropout(dense,params['drop_out'])
    logits = _dense(dense,size_in=params['dense_units'],
                    size_out=10,name='Output',relu=False,log=params['log'])
        
    #### 2 CALCULATIONS AND METRICS
    
    predictions = {"classes": tf.argmax(input=logits,axis=1),
                   "logits": logits,
                   "probabilities": tf.nn.softmax(logits,name='softmax')}
    export_outputs = {'predictions': tf.estimator.export.PredictOutput(predictions)}
    if (mode==tf.estimator.ModeKeys.TRAIN or mode==tf.estimator.ModeKeys.EVAL):
        loss = tf.losses.sparse_softmax_cross_entropy(labels=labels,logits=logits)
        accuracy = tf.metrics.accuracy(
            labels=labels, predictions=tf.argmax(logits,axis=1))
        metrics = {'accuracy':accuracy}
        
    #### 3 MODE = PREDICT
    
    if mode == tf.estimator.ModeKeys.PREDICT:
        return tf.estimator.EstimatorSpec(
            mode=mode, predictions=predictions, export_outputs=export_outputs)

    #### 4 MODE = TRAIN

    if mode == tf.estimator.ModeKeys.TRAIN:
        learning_rate = tf.train.exponential_decay(
            params['learning_rate'],tf.train.get_global_step(),
            decay_steps=100000,decay_rate=0.96)
        optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
        train_op = optimizer.minimize(loss=loss,global_step=tf.train.get_global_step())
        #
        # 
        #
        return tf.estimator.EstimatorSpec(
            mode=mode, loss=loss, train_op=train_op)
    
    #### 5 MODE = EVAL
    
    if mode == tf.estimator.ModeKeys.EVAL:
        return tf.estimator.EstimatorSpec(
            mode=mode,loss=loss,eval_metric_ops=metrics)<jupyter_output><empty_output><jupyter_text>2a - Function to deserialize tfrecords files.Implement parse_tfrecord function:- Feature input: {idx: tf.int64, label: tf.int64, image: tf.string}- Return: image, label<jupyter_code>def parse_tfrecord(example):
    feature={'idx'     : tf.FixedLenFeature((), tf.int64),
             'label'   : tf.FixedLenFeature((), tf.int64),
             'image'   : tf.FixedLenFeature((), tf.string, default_value="")}
    parsed = tf.parse_single_example(example, feature)
    image = tf.decode_raw(parsed['image'],tf.float64)
    image = tf.cast(image,tf.float32)
    image = tf.reshape(image,[32,32,3])
    return image, parsed['label']<jupyter_output><empty_output><jupyter_text>Two helper functions:Implement image_scaling function:- Image scaling.- Applied alwaysImplement distort function:- Applied only in training.- Resize, crop, flip randomly.<jupyter_code>def image_scaling(x):
    return tf.image.per_image_standardization(x)

def distort(x):
    x = tf.image.resize_image_with_crop_or_pad(x, 40, 40)
    x = tf.random_crop(x, [32, 32, 3])
    x = tf.image.random_flip_left_right(x)
    return x<jupyter_output><empty_output><jupyter_text>Input function using tf.dataCreate dataset_input_fn that implements:- Use TFRecordDataset to read tfrecord files.- Apply parse function you created above.- Apply image scaling function you created above.- Apply distort function on for training.- Suffle only for training.- Use prefetch- Optional: parallelize threads wherever possible.<jupyter_code>def dataset_input_fn(params):
    dataset = tf.data.TFRecordDataset(
        params['filenames'],num_parallel_reads=params['threads'])
    dataset = dataset.map(parse_tfrecord, num_parallel_calls=params['threads'])
    dataset = dataset.map(lambda x,y: (image_scaling(x),y),num_parallel_calls=params['threads'])
    if params['mode']==tf.estimator.ModeKeys.TRAIN:
        dataset = dataset.map(lambda x,y: (distort(x),y),num_parallel_calls=params['threads'])
        dataset = dataset.shuffle(buffer_size=params['shuffle_buff'])
    dataset = dataset.repeat()
    dataset = dataset.batch(params['batch'])
    dataset = dataset.prefetch(8*params['batch'])
    return dataset<jupyter_output><empty_output><jupyter_text>DO: Logging Switch ParameterImplement logging parameter to turn on/off histogram logging.<jupyter_code>model_params  = {'drop_out'      : 0.2,
                 'dense_units'   : 1024,
                 'learning_rate' : 1e-3,
                 'log'           : False # change to True
                }<jupyter_output><empty_output><jupyter_text>Create RunConfigImplement RunConfig:- Save checkpoints ever 300 seconds.- Keep up to 5 checkpoint history.<jupyter_code>config = tf.estimator.RunConfig(save_checkpoints_secs = 300,keep_checkpoint_max = 5)
#Set model_fn
model_fn = cnnmodel_fn<jupyter_output><empty_output><jupyter_text>Set model directory.  This is to make sorting out runs more easily.<jupyter_code>name = 'cnn_model/cnn_model_'
name = name + 'dense' + str(model_params['dense_units']) + '_'
name = name + 'drop' + str(model_params['drop_out']) + '_'
name = name + 'lr' + str(model_params['learning_rate']) + '_'
name = name + time.strftime("%Y%m%d%H%M%S")
model_dir  = os.path.join('./',name)

print(model_dir)<jupyter_output>./cnn_model/cnn_model_dense1024_drop0.2_lr0.001_20180731220212<jupyter_text>No Change.<jupyter_code>estimator = tf.estimator.Estimator(
    model_fn=model_fn,model_dir=model_dir,params=model_params,config=config)<jupyter_output>INFO:tensorflow:Using config: {'_master': '', '_save_summary_steps': 100, '_device_fn': None, '_train_distribute': None, '_service': None, '_num_worker_replicas': 1, '_keep_checkpoint_every_n_hours': 10000, '_evaluation_master': '', '_model_dir': './cnn_model/cnn_model_dense1024_drop0.2_lr0.001_20180731220212', '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f98a198afd0>, '_task_type': 'worker', '_session_config': None, '_task_id': 0, '_save_checkpoints_steps': None, '_global_id_in_cluster': 0, '_tf_random_seed': None, '_save_checkpoints_secs': 300, '_log_step_count_steps': 100, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_is_chief': True}<jupyter_text>Set parameters for input functions.<jupyter_code>train_files = !ls ./data/cifar10_data_00*.tfrecords
val_files   = !ls ./data/cifar10_data_01*.tfrecords

train_params = {'filenames'    : train_files,
                'mode'         : tf.estimator.ModeKeys.TRAIN,
                'threads'      : 16,
                'shuffle_buff' : 100000,
                'batch'        : 100
               }

eval_params  = {'filenames'    : val_files,
                'mode'         : tf.estimator.ModeKeys.EVAL,
                'threads'      : 8,
                'batch'        : 200
               }<jupyter_output><empty_output><jupyter_text>DO: TrainSpec and EvalSpecTranSpec:- Use train_params.- End after 20,000 steps.EvalSpec:- Use eval_params.- 10 steps.- Eval executes every 60 seconds or training time (production will be much longer).<jupyter_code>train_spec = tf.estimator.TrainSpec(input_fn=lambda: dataset_input_fn(train_params),max_steps=20000)
eval_spec  = tf.estimator.EvalSpec(input_fn=lambda: dataset_input_fn(eval_params),steps=10,throttle_secs=60)<jupyter_output><empty_output><jupyter_text>No Change.<jupyter_code>tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)<jupyter_output>INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after 60 secs (eval_spec.throttle_secs) or training is finished.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Saving checkpoints for 0 into ./cnn_model/cnn_model_dense1024_drop0.2_lr0.001_20180731220212/model.ckpt.
INFO:tensorflow:loss = 2.3025055, step = 0
INFO:tensorflow:global_step/sec: 77.224
INFO:tensorflow:loss = 2.298281, step = 100 (1.297 sec)
INFO:tensorflow:global_step/sec: 84.0255
INFO:tensorflow:loss = 2.296463, step = 200 (1.190 sec)
INFO:tensorflow:global_step/sec: 83.6519
INFO:tensorflow:loss = 2.2979407, step = 300 (1.195 sec)
INFO:tensorflow:global_step/sec: 84.094
INFO:tensorflow:loss = 2.2817075, step = 400 (1.190 sec)[...]
{'fraction_non_alphanumeric': 0.12870662460567822, 'fraction_numerical': 0.0339291973361374, 'mean_word_length': 4.11326164874552, 'pattern_counts': {'":': 6, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '24439', 'license': 'MIT', 'n_tokens_mistral': 5132, 'n_tokens_neox': 4763, 'n_words': 866, 'path': 'cifar10_tf_keras/03.1a-TensorBoard-Estimator-Graph.ipynb', 'repo_name': 'jasmeetsb/ml_workshop', 'size': '14265'}