<jupyter_start><jupyter_text>Overview - Transitioning to TF 2.0 : Building Preprocessing into the Graph TopicThis notebook demonstrates using the tf.Keras Functional API to add a unique identifier to each prediction when doing a distributed batch prediction using an existing trained model. AudienceThe audience for this notebook are data scientists and machine learning engineers. PrerequistesOne should be familar with:    - TF.Keras    - TF 2.0 Subclassing & Decorators    - Data Preprocessing     DatasetThis notebook uses the builtin dataset in tf.Keras for MNIST.This dataset contains train and test images for training an image classifier to recognize handwritten digits. ObjectiveWe will be using the following TF 2.0 features / recommendations:    1. [Recommentation] Use tf.keras for the model building.    2. [Recommendation] Put preprocessing into the model.    3. [Feature] Use @tf.function decorator to convert the Python code for preprocessing into graph ops.    4. [Feature] Use subclassing of layers to define a new layer for the preprocessing. Costs This notebook can be ran either locally on your laptop/PC or free on Colab. Set up your local development environment**If you are using Colab or AI Platform Notebooks**, with the exception of TF 2.0, your environment already meetsall the remaining requirements to run this notebook. You can skip this step.**Otherwise**, make sure your environment meets this notebook's requirements.You need the following:* Git* Python 3* Jupyter notebook running in a virtual environment with Python 3* Tensorflow 2.0* Keras  Tutorial Import Libraries and Define ConstantsIf you haven't already, you need to install TF 2.0 beta. If you are running this notebook in colab (which is 1.13 as of this writing), you will need to install TF 2.0 via a cell, as in below:<jupyter_code># If not already installed
%pip install tensorflow==2.0.0-beta1<jupyter_output><empty_output><jupyter_text>Now let's import what we will use in this demonstration.<jupyter_code>import tensorflow as tf
from tensorflow.keras import Model, Input, layers
from tensorflow.keras.layers import Flatten, Dense

# expected output: 2.0.0-beta1
print(tf.__version__)<jupyter_output><empty_output><jupyter_text>Layer SubclassingWe will start by subclassing the tf.keras layers class to make a new layer type, as follows:    1. Will take an input vector whose shape is specified at instantiation.    2. Will normalize the data between 0 and 1 (assumes pixel data between 0 .. 255).    3. Outputs the normalized input.    4. Has no trainable parameters.    Let's start by showing a basic template for subclassing layers and then explain it:```pythonclass NewLayer(layers.Layer):    def __init__(self):        super(NewLayer, self).__init__()        self.my_vars = blash, blah        def build(self, input_shape):        """ Handler for building the layer """        self.kernel = blah, blah        def call(self, inputs):        """ Handler for layer object as callable """        outputs = do something with inputs        return outputs``` SubclassingThe first line in the above template `class NewLayer(layers.Layer)` indicates we want to create a new class object named `NewLayer` which is subclassed (derived) from the tf.keras `layers` class. This will give us a custom layer definition. __init__() methodThis is the initializer (constructor) for the class object instantiation. We use the initializer to initialize layer specific variables. build() methodThis method handles the building of the layer when the model is compiled. A typical action is to define the shape of the kernel (trainable parameters) and initialization of the kernel. call() methodThis method handles calling the layer as a callable (function call) for execution in the graph.  Subclassing Our Custom LayerIn the code below, we subclass a custom layer for doing preprocessing of the input, and where the preprocessing is converted to graph operations in the model.The first line in the code `class Normalize(layers.Layer)` indicates we want to create a new class object named `Normalize` which is subclassed (derived) from the tf.keras `layers` class.  __init__() methodSince we won't have any constants or variables to preserve, we don't have any need to add anything to this method. build() methodOur custom layer won't have any trainable parameters. We will tell the compile process to not set up any gradient descent updates on the kernel during training by setting the `layers` class variable `self.kernel` to `None`. call() methodThis is where we add our preprocessing. The parameter `inputs` is the input tensor to the layer during training and prediction. A TF tensor object implements polymorphism to overload operators. We use the overloaded division operator, which will broadcast the division operation across the entire tensor --thus each element will be divided by 255.0.Finally, we add the decorator `@tf.function` to tell **TensorFlow AutoGraph** to convert convert the Python code in this method to graph operations in the model.<jupyter_code>class Normalize(layers.Layer):
    """ Custom Layer for Preprocessing Input """
    def __init__(self):
        """ Constructor """
        super(Normalize, self).__init__()
    
    def build(self, input_shape):
        """ Handler for Input Shape """
        self.kernel = None
    
    @tf.function
    def call(self, inputs):
        """ Handler for layer object is callable """
        inputs = inputs / 255.0
        return inputs<jupyter_output><empty_output><jupyter_text>Build the ModelLet's build a model to train on the MNIST dataset. We will keep it really basic:    1. Use the Functional API method for defining the model.    2. Make the first layer of our model the custom preprocessing layer.    3. The remaining layers are a basic DNN for MNIST.<jupyter_code># Create the input vector for 28x28 MNIST images
inputs = Input((28, 28))

# The first layer is the preprocessing layer, which is bound to the input vector
x = Normalize()(inputs)

# Next layer, we flatten the preprocessed input into a 1D vector
x = Flatten()(x)

# Create a hidden dense layer of 128 nodes
x = Dense(128, activation='relu')(x)

# Create an output layer for classifying the 10 digits
outputs = Dense(10, activation='sigmoid')(x)

# Instantiate the model
model = Model(inputs, outputs)

# Compile the model
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])<jupyter_output><empty_output><jupyter_text>Prepare the DataWe will get the tf.keras builtin dataset for MNIST. The dataset is pre-split into train and test data. The data is separated into numpy multi-dimensional arrays for images and labels. The image data is not preprocessed --i.e., all the values are between 0 and 255. The label data is not one-hot-encoded --hence why we compiled with `loss='sparse_categorical_crossentropy'`<jupyter_code>from tensorflow.keras.datasets import mnist

# Load the train and test data into memory
(x_train, y_train), (x_test, y_test) = mnist.load_data()

# Expected output: (60000, 28, 28) , (60000,)
print(x_train.shape, y_train.shape)<jupyter_output><empty_output><jupyter_text>Train the ModelLet's now train the model (with the preprocessing built into the model graph) on the unpreprocessed MNIST data.<jupyter_code>model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.1, verbose=1)<jupyter_output><empty_output><jupyter_text>Evaluate the ModelLet's now evaluate (prediction) using unpreprocessed test examples on the trained model.<jupyter_code>acc = model.evaluate(x_test, y_test)
print(acc)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06650924832743015, 'fraction_numerical': 0.012593467138921684, 'mean_word_length': 4.585347985347985, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '187264', 'license': 'Apache-2.0', 'n_tokens_mistral': 2083, 'n_tokens_neox': 1953, 'n_words': 981, 'path': 'notebooks/ai_hub/ai_hub_building_preprocessing_into_graph.ipynb', 'repo_name': 'thedriftofwords/keras-idiomatic-programmer', 'size': '7623'}