<jupyter_start><jupyter_text>3 - Build models with the Keras API Motivation- **`tf.keras`** is the high-level TensorFlow API meant to create layers and models.- With `tf.keras`, you will be able to build any machine learning model, as well as your own model. Objectives- Build your first model with `tf.keras.Sequential`- Build complex models with the Functional API- Discover TensorFlow's built-in layers and models- Make custom layers and models from scratch- Learn how to do  transfer learning with TFHub<jupyter_code>import tensorflow as tf
import numpy as np
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow_hub as hub
tf.__version__<jupyter_output><empty_output><jupyter_text>Table of contents:*  [Overview](overview)*  [Part 1: Build simple models with `tf.keras.Sequential`](part-1)    *  [1.1 Keras's built-in layers](1.1)    *  [1.2 Your first sequential model](1.2)    *  [1.3 Specify input's shape](1.3)    *  [1.4 Vizualize your model with `tf.keras.Model.summary()`](1.4)    *  [1.5 Exercise](1.5)*  [Part 2: Build complex models with the functional API](part-2)    *  [2.1 Your first model with the functional API](2.1)    *  [2.2 Create complex model architecture](2.2)        * [2.2.1 Multiple inputs and outputs models](2.2.1)        * [2.2.2 Residual Neural Networks](2.2.2)        * [2.2.3 Autoencoders](2.2.3)    *  [2.3 Exercise](2.3)*  [Part 3: Create custom layers and models](part-3)    *  [3.1 Build custom layers](3.1)    *  [3.2 Build custom models](3.2)*  [Part 4: Transfer learning with TensorFlow Hub](part-4)*  [Summary](summary)*  [Where to go next](next) Overview - `tf.keras` is the high-level TensorFlow API meant to create layers and models.- `tf.keras.layers` contains most layers needed to build most deep learning models.- `tf.keras.Sequential` is used to stack `tf.keras.layers` sequentially and to build simple models.- `tf.keras.Model` is used to build complex models with many inputs and many outputs.- If `tf.keras.Model` isn't flexible enough for you, you can inherit from `tf.keras.layers.Layer` and `tf.keras.Model` to build custom class layers and class models. Part 1:  Build simple models with `tf.keras.Sequential` - Let's build our first model with `tf.keras.Sequential`. 1.1 Keras's built-in layers **`tf.keras.layers`** includes a wide range of built-in layers, for example:- ANN layers: Dense- Convolutional layers: Conv1D, Conv2D, Conv3D, Conv2DTranspose- Pooling layers: MaxPooling1D, MaxPooling2D, MaxPooling3D, AveragePooling1D- RNN layers: RNN, GRU, LSTM- Dropout, BatchNormalization, Embedding, PositionalEncoding, etc.Layers have to be defined with its output size, and optionally its activation function, and its kernel and bias initializers.<jupyter_code># Define a dense layer having an output space's dimension of 2
layer = layers.Dense(units=2, activation="relu")<jupyter_output><empty_output><jupyter_text>Many activation functions are already implemented in TensorFlow. You can find them in [`tf.keras.activations`](https://www.tensorflow.org/api_docs/python/tf/keras/activations).<jupyter_code># All layers are callable
inputs = tf.random.uniform((5, 3))
layer(inputs)<jupyter_output><empty_output><jupyter_text>**⚠️Caveat:** Layers create its weights the first time it is called on an input. Besides, layers attribute cannot be modified after the layer has been called once. The following leads to an ValueError:<jupyter_code># ValueError:
# inputs = tf.random.uniform((5, 2))
# layer(inputs)<jupyter_output><empty_output><jupyter_text>Indeed, after being called `layer` has now a weight matrix of shape `(3, 2)` and can only take inputs of shape `(None, 3)`.**Note:** in TensorFlow `None` in a shape is for _"any dimension"_.  1.2 Your first sequential model Now that we know more about `tf.keras.layers`, let's build a sequential model:<jupyter_code># Create a Sequential model by passing a list of layers
model = keras.Sequential(
    [
        layers.Dense(32, activation="relu"),
        layers.Dense(16, activation="relu"),
        layers.Dense(4),
    ]
)
inputs = tf.random.uniform((5, 10))
# Output is of shape (5, 4)
model(inputs)<jupyter_output><empty_output><jupyter_text>You can also create a Sequential model incrementally via the `add()` method.<jupyter_code># Other way to define a sequential model:
model = keras.Sequential()
model.add(layers.Dense(32, activation="relu"))
model.add(layers.Dense(16, activation="relu"))
model.add(layers.Dense(4, activation="relu"))
inputs = tf.random.uniform((5, 10))
# Output is of shape (5, 4)
model(inputs)<jupyter_output><empty_output><jupyter_text>1.3 Specify input's shape - If you want to specify the input shape of your model before calling it, you can use the layer `keras.Input` as your model's first layer.- When you specify your input's shape, the batch size can be omitted.- For example, if you have an image input with a shape of `(32, 32, 3)`, you should use `keras.Input(shape=(32, 32, 3))`.- Alternatively, you can use the `input_shape` in when defining your model's first layer.<jupyter_code>model = keras.Sequential(
    [
        keras.Input(shape=(10,)),
        layers.Dense(32, activation="relu"),
        layers.Dense(16, activation="relu"),
        layers.Dense(4),
    ]
)
# Other manner to specify the input's shape
layers.Dense(32, activation="relu", input_shape=(10,))
inputs = tf.random.uniform((5, 10))
# Output is of shape (5, 4)
model(inputs)<jupyter_output><empty_output><jupyter_text>1.4 Vizualize your model with `tf.keras.Model.summary()` Note that Layers accepts a name argument in Keras. This is useful when visualizing your model with the `summary()` method.<jupyter_code># Same model with name in layers
model = keras.Sequential(
    [
        keras.Input(shape=(10,), name='input'),
        layers.Dense(32, activation="relu", name='hidden1'),
        layers.Dense(16, activation="relu", name='hidden2'),
        layers.Dense(4, name='output'),
    ]
)
# Model's summary
model.summary()<jupyter_output>Model: "sequential_5"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
hidden1 (Dense)              (None, 32)                352       
_________________________________________________________________
hidden2 (Dense)              (None, 16)                528       
_________________________________________________________________
output (Dense)               (None, 4)                 68        
=================================================================
Total params: 948
Trainable params: 948
Non-trainable params: 0
_________________________________________________________________<jupyter_text>You can plot your model with `keras.utils.plot_model` (needs pydot and graphviz).<jupyter_code># Plot model
keras.utils.plot_model(model)<jupyter_output><empty_output><jupyter_text>1.5 Exercise ![fashion_mnist](./ressources/fashion_mnist.png)Define an artificial neural network to classify the **fashion_mnist** dataset. fashion_mnist is a dataset of Zalando's article images. Each example is a 28x28 grayscale image, associated with a label from 10 classes.The model architecture will be:- a 'Flatten' layer taking an input of shape $28x28$.- A Dense layer having 128 as output's dimension and 'relu' for activation.- A 'Dropout' layer having 0.2 for rate.- A Dense layer having 10 as output's dimension. Print the summary of this model.<jupyter_code>model = tf.keras.Sequential([
    layers.Flatten(input_shape=(28, 28)),
    layers.Dense(128, activation='relu'),
    layers.Dropout(0.2),
    layers.Dense(10)
])
model.summary()<jupyter_output>Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten (Flatten)            (None, 784)               0         
_________________________________________________________________
dense_11 (Dense)             (None, 128)               100480    
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
dense_12 (Dense)             (None, 10)                1290      
=================================================================
Total params: 101,770
Trainable params: 101,770
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Part 2: Build complex models with the functional API - The Keras **functional API** is a way to create models that are more flexible than the `tf.keras.Sequential` API.- Defining a model via the functional API is like defining nodes in a graph of layers. 2.1  Your first model with the functional API Let's rebuild the model from the last exercise using the Functional API. Start by creating an input node with `keras.Input`:<jupyter_code>inputs = keras.Input((28, 28))<jupyter_output><empty_output><jupyter_text>Then, create a new layer and pass the input to this layer by calling the new layer on `x`:<jupyter_code>flatten_layer = layers.Flatten(input_shape=(28, 28))
x = flatten_layer(inputs)<jupyter_output><empty_output><jupyter_text>The "layer call" action is like drawing an arrow from "inputs" to this layer you created.Let's add a few more layers to the graph of layers:<jupyter_code>x = layers.Dense(128, activation="relu")(x)
x = layers.Dropout(0.2)(x)
outputs = layers.Dense(10)(x)<jupyter_output><empty_output><jupyter_text>Finally, create a `keras.Model` by specifying its inputs and outputs:<jupyter_code>model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")
model.summary()<jupyter_output>Model: "mnist_model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 28, 28)]          0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 784)               0         
_________________________________________________________________
dense_13 (Dense)             (None, 128)               100480    
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_14 (Dense)             (None, 10)                1290      
=================================================================
Total params: 101,770
Trainable params: 101,770
Non-trainable params: 0
_________________________________________________[...]<jupyter_text>In the end, the functional API is an intuitive way to create models  as a graphs of layers.  2.2  Create complex model architecture The functional API can handle models with multiple inputs or outputs, shared layers, and even non-linear topology. All of these cannot be handled with the Sequential API. 2.2.1 Multiple inputs and outputs models The functional API is convenient enough to manipulate multiple inputs and outputs.  For example, the following model takes two inputs:- a movie's title- a short summary of that movie And it should predict two outputs:- that movie's category (14 classes: Sci-fi & Fantasy, Comedy, Romantic, etc)- its rating by Netflix users. ![multi_inputs-outputs_model.png](ressources/multi_inputs-outputs_model.png)This is how you do it with the functional API:<jupyter_code>vocab_size = 10000  # Size of vocabulary obtained when preprocessing text data

title_input = keras.Input(shape=(None,), name="title")
summary_input = keras.Input(shape=(None,), name="summary")

# Embed each word in the title into a 64-dimensional vector
title_features = layers.Embedding(vocab_size, 64)(title_input)
# Embed each word in the summary into a 64-dimensional vector
summary_features = layers.Embedding(vocab_size, 64)(summary_input)

# Reduce sequence of embedded words in the title into a single 32-dimensional vector
title_features = layers.LSTM(32)(title_features)
# Reduce sequence of embedded words in the summary into a single 128-dimensional vector
summary_features = layers.LSTM(128)(summary_features)

# Merge all available features into a single large vector via concatenation
x = layers.concatenate([title_features, summary_features])

# Category classifier
category_pred = layers.Dense(14, name="category")(x)
# Logistic regression for rating prediction
rating_pred = layers.Dense(1, name="rating")(x)
# Instantiate an end-to-end model predicting both priority and department
model = keras.Model(
    inputs=[title_input, summary_input],
    outputs=[category_pred, rating_pred],
)<jupyter_output><empty_output><jupyter_text>2.2.2 Residual Neural Networks  Residual neural networks (Resnets) are neural networks with residual connections that jump over some layers (as shown in the following figure). ![residual_connection.png](./ressources/residual_connection.png)They are very popular models for image classication. The following model as shown in the image below (_right-click "open image in a new tab" to zoom in_) is an implementation of a resnet for image classification:![resnet.png](./ressources/resnet.png)<jupyter_code>inputs = keras.Input(shape=(32, 32, 3), name="img")
x = layers.Conv2D(32, 3, activation="relu")(inputs)
x = layers.Conv2D(64, 3, activation="relu")(x)
block_1_output = layers.MaxPooling2D(3)(x)

x = layers.Conv2D(64, 3, activation="relu", padding="same")(block_1_output)
x = layers.Conv2D(64, 3, activation="relu", padding="same")(x)
block_2_output = layers.add([x, block_1_output])

x = layers.Conv2D(64, 3, activation="relu", padding="same")(block_2_output)
x = layers.Conv2D(64, 3, activation="relu", padding="same")(x)
block_3_output = layers.add([x, block_2_output])

x = layers.Conv2D(64, 3, activation="relu")(block_3_output)
x = layers.GlobalAveragePooling2D()(x)
x = layers.Dense(256, activation="relu")(x)
x = layers.Dropout(0.5)(x)
outputs = layers.Dense(10, name="outputs")(x)

model = keras.Model(inputs, outputs)<jupyter_output><empty_output><jupyter_text>2.2.3  Autoencoders An autoencoder is a neural network that is used to learn a condensed representation (encoding) of original inputs.Autoencoders are used for dimensionality reduction or denoising. Typically, autoencoders impose a bottleneck in the network to learn a condensed representation of the original data.Since the functional API enables you to use a single stack of layers to generate multiple models, autoencoders are easy to build with the functional API. For example, the following model is an autoencoder built using the same stack of layers to instantiate two models: an encoder model that turns image inputs into a 16-dimensional vectors, and an end-to-end autoencoder model built for image denoising.**Encoder:**![encoder.png](./ressources/encoder.png)**Autoencoder:**![autoencoder.png](./ressources/autoencoder.png)<jupyter_code>encoder_input = keras.Input(shape=(28, 28, 1), name="img")
x = layers.Conv2D(16, 3, activation="relu")(encoder_input)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.MaxPooling2D(3)(x)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.Conv2D(16, 3, activation="relu")(x)
encoder_output = layers.GlobalMaxPooling2D(name='encoder_output')(x)

encoder = keras.Model(encoder_input, encoder_output, name="encoder")
x = layers.Reshape((4, 4, 1))(encoder_output)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
x = layers.Conv2DTranspose(32, 3, activation="relu")(x)
x = layers.UpSampling2D(3)(x)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
decoder_output = layers.Conv2DTranspose(1, 3, activation="relu", name='decoder_output')(x)

autoencoder = keras.Model(encoder_input, decoder_output, name="autoencoder")<jupyter_output><empty_output><jupyter_text>2.3: Exercise Build the following model with the functional API:![exercise_2_3.png](./ressources/exercise_2_3.png)Notice that `inputs1` and `inputs2` are fed to the same layer.<jupyter_code>inputs1 = keras.Input(shape=(28,), name="inputs1")
inputs2 = keras.Input(shape=(28,), name="inputs2")
common_layer = layers.Dense(10, activation='relu')

x1 = common_layer(inputs1)
x2 = common_layer(inputs2)
x = layers.Concatenate()([x1, x2])
outputs = layers.Dense(20, activation='relu')(x)
outputs = layers.Add(name="outputs")([x, outputs])

model = keras.Model([inputs1, inputs2], outputs)<jupyter_output><empty_output><jupyter_text>Part 3: Create custom layers and models The Sequential API and functional API are flexible enough to allow you to build almost any existing machine learning model. However, if you can't find the layer or model you need for your project, Keras allows you to build custom layers and models simply by subclassing `tf.keras.layers.Layer` and `tf.keras.Model`. 3.1 Build custom layers  To create your own layer, subclass `tf.keras.Layer` class and implement:1. an `__init__` method, where you set your layer parameters.1. a `build` method, where you know the shapes of the input tensors and creates the weights of the layer.1. a `call` method, that specifies the computation done by the layer.1. a `get_config` method, that returns a dictionary of the arguments used to create the layer. This method is required only if you want your layer to be serializable.You can build your layer in the `__init__` method and thus not implement the build method. However, in this case input shapes must be given during the `__init__`. The following example, recreates `layers.Dense` from scratch.<jupyter_code># Custom Dense layer by sublclassing layers.Layer
class CustomDense(layers.Layer):
    def __init__(self, units):
        super().__init__()
        self.units = units

    def build(self, input_shape):
        self.w = self.add_weight(
            shape=(input_shape[-1], self.units),
            initializer="random_normal",
            trainable=True,
        )
        self.b = self.add_weight(
            shape=(self.units,), initializer="random_normal", trainable=True
        )

    def call(self, inputs):
        return tf.matmul(inputs, self.w) + self.b
    
    def get_config(self):
        return {"units": self.units}
# Work as any other layer
layer = CustomDense(units=5)
inputs = tf.random.uniform((2, 3))
layer(inputs)<jupyter_output><empty_output><jupyter_text>3.2 Build custom models  First, let's list the difference between `keras.Model` and `keras.layers.Layer`:`keras.Model` exposes built-in training, evaluation, and prediction loops (`model.fit()`, `model.evaluate()`, `model.predict()`).`keras.Model` exposes the list of its inner layers, via the `model.layers` property.`keras.Model` exposes saving and serialization APIs (`save()`, `save_weights()`, etc)If you want to build a custom model, subclass `tf.keras.Model` class and implement:- an `__init__` method, that defines all the layers needed.- a `call` method, that build the graph of layers in the same way as the functional API.The following example illustrates this by creating an<jupyter_code>class MLP(keras.Model):

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.dense_1 = layers.Dense(64, activation='relu')
        self.dense_2 = layers.Dense(10, activation='softmax')

    def call(self, inputs):
        x = self.dense_1(inputs)
        return self.dense_2(x)
# Work as any other model
model = MLP(name='mlp')
inputs = tf.random.uniform((2, 3))
model(inputs)
model.summary()<jupyter_output>Model: "mlp"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_51 (Dense)             multiple                  256       
_________________________________________________________________
dense_52 (Dense)             multiple                  650       
=================================================================
Total params: 906
Trainable params: 906
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Part 4: Transfer learning with TensorFlow Hub - Transfer learning consists of taking a model already trained for a certain problem, and using it for a new, similar problem.  - For example, a model that has been trained to classify tree flowers may be useful to build a model meant to identify wild flower.[TensorFlow Hub](https://tfhub.dev/) is a repository of pre-trained machine learning models ready for to be fine-tuned to your project. With TF Hub, you can reuse trained models like BERT, MobileNet for image classification, and many other ML models.To load a pre-trained model with TensorFlow Hub, find its url from tfhub.dev and add it into the hub.KerasLayer API:<jupyter_code># MobileNet V2 model
model_url = "https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/5"

# turn its variables to non-trainable variables
mobile_net = hub.KerasLayer(model_url, trainable=False)

model = tf.keras.Sequential([
    mobile_net,
    layers.Dense(10)
])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12295390404515522, 'fraction_numerical': 0.02464722483537159, 'mean_word_length': 4.7868807838867715, 'pattern_counts': {'":': 1, '<': 81, '<?xml version=': 0, '>': 81, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '405179', 'license': 'Apache-2.0', 'n_tokens_mistral': 6354, 'n_tokens_neox': 5920, 'n_words': 2048, 'path': '3 - Build models with the Keras API.ipynb', 'repo_name': 'oyiakoumis/tensorflow2-course', 'size': '21260'}