<jupyter_start><jupyter_text><jupyter_code>!pip install tensorflow==2.0.0
import tensorflow as tf
print(tf.__version__)<jupyter_output>2.0.0<jupyter_text>TensorFlow 2.0 + Keras Overview for Deep Learning Researchers*@fchollet, October 2019*---**This document serves as an introduction, crash course, and quick API reference for TensorFlow 2.0.**---TensorFlow and Keras were both released over four years ago (March 2015 for Keras and November 2015 for TensorFlow). That's a long time in deep learning years!In the old days, TensorFlow 1.x + Keras had a number of known issues:- Using TensorFlow meant manipulating static computation graphs, which would feel awkward and difficult to programmers used to imperative styles of coding.- While the TensorFlow API was very powerful and flexible, it lacked polish and was often confusing or difficult to use.- While Keras was very productive and easy to use, it would often lack flexibility for research use cases.--- TensorFlow 2.0 is an extensive redesign of TensorFlow and Keras that takes into account over four years of user feedback and technical progress. It fixes the issues above in a big way. It's a machine learning platform from the future.---TensorFlow 2.0 is built on the following key ideas:- Let users run their computation eagerly, like they would in Numpy. This makes TensorFlow 2.0 programming intuitive and Pythonic.- Preserve the considerable advantages of compiled graphs (for performance, distribution, and deployment). This makes TensorFlow fast, scalable, and production-ready.- Leverage Keras as its high-level deep learning API, making TensorFlow approachable and highly productive.- Extend Keras into a spectrum of workflows ranging from the very high-level (easier to use, less flexible) to the very low-level (requires more expertise, but provides great flexibility).  Part 1: TensorFlow basics  Tensors This is a [constant](https://www.tensorflow.org/api_docs/python/tf/constant) tensor:<jupyter_code>x = tf.constant([[5, 2], [1, 3]])
print(x)<jupyter_output>tf.Tensor(
[[5 2]
 [1 3]], shape=(2, 2), dtype=int32)<jupyter_text>You can get its value as a Numpy array by calling `.numpy()`:<jupyter_code>x.numpy()<jupyter_output><empty_output><jupyter_text>Much like a Numpy array, it features the attributes `dtype` and `shape`:<jupyter_code>print('dtype:', x.dtype)
print('shape:', x.shape)<jupyter_output>dtype: <dtype: 'int32'>
shape: (2, 2)<jupyter_text>A common way to create constant tensors is via `tf.ones` and `tf.zeros` (just like `np.ones` and `np.zeros`):<jupyter_code>print(tf.ones(shape=(2, 1)))
print(tf.zeros(shape=(2, 1)))<jupyter_output>tf.Tensor(
[[1.]
 [1.]], shape=(2, 1), dtype=float32)
tf.Tensor(
[[0.]
 [0.]], shape=(2, 1), dtype=float32)<jupyter_text>Random constant tensors This is all pretty [normal](https://www.tensorflow.org/api_docs/python/tf/random/normal):<jupyter_code>tf.random.normal(shape=(2, 2), mean=0., stddev=1.)<jupyter_output><empty_output><jupyter_text>And here's an integer tensor with values drawn from a random [uniform](https://www.tensorflow.org/api_docs/python/tf/random/uniform) distribution:<jupyter_code>tf.random.uniform(shape=(2, 2), minval=0, maxval=10, dtype='int32')<jupyter_output><empty_output><jupyter_text>Variables [Variables](https://www.tensorflow.org/guide/variable) are special tensors used to store mutable state (like the weights of a neural network). You create a Variable using some initial value.<jupyter_code>initial_value = tf.random.normal(shape=(2, 2))
a = tf.Variable(initial_value)
print(a)<jupyter_output><tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=
array([[-0.16258094, -0.52607477],
       [ 0.69424504,  2.1672049 ]], dtype=float32)><jupyter_text>You update the value of a Variable by using the methods `.assign(value)`, or `.assign_add(increment)` or `.assign_sub(decrement)`:<jupyter_code>new_value = tf.random.normal(shape=(2, 2))
a.assign(new_value)
for i in range(2):
  for j in range(2):
    assert a[i, j] == new_value[i, j]
added_value = tf.random.normal(shape=(2, 2))
a.assign_add(added_value)
for i in range(2):
  for j in range(2):
    assert a[i, j] == new_value[i, j] + added_value[i, j]<jupyter_output><empty_output><jupyter_text>Doing math in TensorFlow You can use TensorFlow exactly like you would use Numpy. The main difference is that your TensorFlow code can run on GPU and TPU.<jupyter_code>a = tf.random.normal(shape=(2, 2))
b = tf.random.normal(shape=(2, 2))

c = a + b
d = tf.square(c)
e = tf.exp(d)<jupyter_output><empty_output><jupyter_text>Computing gradients with `GradientTape` Oh, and there's another big difference with Numpy: you can automatically retrieve the gradient of any differentiable expression.Just open a [`GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape), start "watching" a tensor via `tape.watch()`, and compose a differentiable expression using this tensor as input:<jupyter_code>a = tf.random.normal(shape=(2, 2))
b = tf.random.normal(shape=(2, 2))

with tf.GradientTape() as tape:
  tape.watch(a)  # Start recording the history of operations applied to `a`
  c = tf.sqrt(tf.square(a) + tf.square(b))  # Do some math using `a`
  # What's the gradient of `c` with respect to `a`?
  dc_da = tape.gradient(c, a)
  print(dc_da)<jupyter_output>tf.Tensor(
[[0.60742545 0.39205843]
 [0.73241967 0.11925844]], shape=(2, 2), dtype=float32)<jupyter_text>By default, variables are watched automatically, so you don't need to manually `watch` them:<jupyter_code>a = tf.Variable(a)

with tf.GradientTape() as tape:
  c = tf.sqrt(tf.square(a) + tf.square(b))
  dc_da = tape.gradient(c, a)
  print(dc_da)<jupyter_output>tf.Tensor(
[[0.60742545 0.39205843]
 [0.73241967 0.11925844]], shape=(2, 2), dtype=float32)<jupyter_text>Note that you can compute higher-order derivatives by nesting tapes:<jupyter_code>with tf.GradientTape() as outer_tape:
  with tf.GradientTape() as tape:
    c = tf.sqrt(tf.square(a) + tf.square(b))
    dc_da = tape.gradient(c, a)
  d2c_da2 = outer_tape.gradient(dc_da, a)
  print(d2c_da2)<jupyter_output>tf.Tensor(
[[0.5418279  0.68627995]
 [0.3451818  0.3664849 ]], shape=(2, 2), dtype=float32)<jupyter_text>An end-to-end example: linear regression So far you've learned that TensorFlow is a Numpy-like library that is GPU or TPU accelerated, with automatic differentiation. Time for an end-to-end example: let's implement a linear regression, the FizzBuzz of Machine Learning. For the sake of demonstration, we won't use any of the higher-level Keras components like `Layer` or `MeanSquaredError`. Just basic ops.<jupyter_code>input_dim = 2
output_dim = 1
learning_rate = 0.01

# This is our weight matrix
w = tf.Variable(tf.random.uniform(shape=(input_dim, output_dim)))
# This is our bias vector
b = tf.Variable(tf.zeros(shape=(output_dim,)))

def compute_predictions(features):
  return tf.matmul(features, w) + b

def compute_loss(labels, predictions):
  return tf.reduce_mean(tf.square(labels - predictions))

def train_on_batch(x, y):
  with tf.GradientTape() as tape:
    predictions = compute_predictions(x)
    loss = compute_loss(y, predictions)
    # Note that `tape.gradient` works with a list as well (w, b).
    dloss_dw, dloss_db = tape.gradient(loss, [w, b])
  w.assign_sub(learning_rate * dloss_dw)
  b.assign_sub(learning_rate * dloss_db)
  return loss<jupyter_output><empty_output><jupyter_text>Let's generate some artificial data to demonstrate our model:<jupyter_code>import numpy as np
import random
import matplotlib.pyplot as plt
%matplotlib inline

# Prepare a dataset.
num_samples = 10000
negative_samples = np.random.multivariate_normal(
    mean=[0, 3], cov=[[1, 0.5],[0.5, 1]], size=num_samples)
positive_samples = np.random.multivariate_normal(
    mean=[3, 0], cov=[[1, 0.5],[0.5, 1]], size=num_samples)
features = np.vstack((negative_samples, positive_samples)).astype(np.float32)
labels = np.vstack((np.zeros((num_samples, 1), dtype='float32'),
                    np.ones((num_samples, 1), dtype='float32')))

plt.scatter(features[:, 0], features[:, 1], c=labels[:, 0])<jupyter_output><empty_output><jupyter_text>Now let's train our linear regression by iterating over batch-by-batch over the data and repeatedly calling `train_on_batch`:<jupyter_code># Shuffle the data.
indices = np.random.permutation(len(features))
features = features[indices]
labels = labels[indices]

# Create a tf.data.Dataset object for easy batched iteration
dataset = tf.data.Dataset.from_tensor_slices((features, labels))
dataset = dataset.shuffle(buffer_size=1024).batch(256)

for epoch in range(10):
  for step, (x, y) in enumerate(dataset):
    loss = train_on_batch(x, y)
  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))<jupyter_output>Epoch 0: last batch loss = 0.0777
Epoch 1: last batch loss = 0.0337
Epoch 2: last batch loss = 0.0326
Epoch 3: last batch loss = 0.0287
Epoch 4: last batch loss = 0.0334
Epoch 5: last batch loss = 0.0261
Epoch 6: last batch loss = 0.0307
Epoch 7: last batch loss = 0.0155
Epoch 8: last batch loss = 0.0230
Epoch 9: last batch loss = 0.0205<jupyter_text>Here's how our model performs:<jupyter_code>predictions = compute_predictions(features)
plt.scatter(features[:, 0], features[:, 1], c=predictions[:, 0] > 0.5)<jupyter_output><empty_output><jupyter_text>Making it fast with `tf.function` But how fast is our current code running?<jupyter_code>import time

t0 = time.time()
for epoch in range(20):
  for step, (x, y) in enumerate(dataset):
    loss = train_on_batch(x, y)
t_end = time.time() - t0
print('Time per epoch: %.3f s' % (t_end / 20,))<jupyter_output>Time per epoch: 0.140 s<jupyter_text>Let's compile the training function into a static graph. Literally all we need to do is add the `tf.function` decorator on it:<jupyter_code>@tf.function
def train_on_batch(x, y):
  with tf.GradientTape() as tape:
    predictions = compute_predictions(x)
    loss = compute_loss(y, predictions)
    dloss_dw, dloss_db = tape.gradient(loss, [w, b])
  w.assign_sub(learning_rate * dloss_dw)
  b.assign_sub(learning_rate * dloss_db)
  return loss<jupyter_output><empty_output><jupyter_text>Let's try this again:<jupyter_code>t0 = time.time()
for epoch in range(20):
  for step, (x, y) in enumerate(dataset):
    loss = train_on_batch(x, y)
t_end = time.time() - t0
print('Time per epoch: %.3f s' % (t_end / 20,))<jupyter_output>Time per epoch: 0.085 s<jupyter_text>40% reduction, neat. In this case we used a trivially simple model; in general the bigger the model the greater the speedup you can get by leveraging static graphs.Remember: eager execution is great for debugging and printing results line-by-line, but when it's time to scale, static graphs are a researcher's best friends.  Part 2: The Keras API Keras is a Python API for deep learning. It has something for everyone:- If you're an engineer, Keras provides you with reusable blocks such as layers, metrics, training loops, to support common use cases. It provides a high-level user experience that's accessible and productive.- If you're a researcher, you may prefer not to use these built-in blocks such as layers and training loops, and instead create your own. Of course, Keras allows you to do this. In this case, Keras provides you with templates for the blocks you write, it provides you with structure, with an API standard for things like Layers and Metrics. This structure makes your code easy to share with others and easy to integrate in production workflows.- The same is true for library developers: TensorFlow is a large ecosystem. It has many different libraries. In order for different libraries to be able to talk to each other and share components, they need to follow an API standard. That's what Keras provides.Crucially, Keras brings high-level UX and low-level flexibility together fluently: you no longer have on one hand, a high-level API that's easy to use but inflexible, and on the other hand a low-level API that's flexible but only approachable by experts. Instead, you have a spectrum of workflows, from the  very high-level to the very low-level. Workflows that are all compatible because they're built on top of the same concepts and objects.![Spectrum of Keras workflows](https://keras-dev.s3.amazonaws.com/tutorials-img/spectrum-of-workflows.png)  The base `Layer` classThe first class you need to know is [`Layer`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer). Pretty much everything in Keras derives from it.A Layer encapsulates a state (weights) and some computation (defined in the `call` method).<jupyter_code>from tensorflow.keras.layers import Layer

class Linear(Layer):
  """y = w.x + b"""

  def __init__(self, units=32, input_dim=32):
      super(Linear, self).__init__()
      w_init = tf.random_normal_initializer()
      self.w = tf.Variable(
          initial_value=w_init(shape=(input_dim, units), dtype='float32'),
          trainable=True)
      b_init = tf.zeros_initializer()
      self.b = tf.Variable(
          initial_value=b_init(shape=(units,), dtype='float32'),
          trainable=True)

  def call(self, inputs):
      return tf.matmul(inputs, self.w) + self.b

# Instantiate our layer.
linear_layer = Linear(4, 2)<jupyter_output><empty_output><jupyter_text>A layer instance works like a function. Let's call it on some data:<jupyter_code>y = linear_layer(tf.ones((2, 2)))
assert y.shape == (2, 4)<jupyter_output><empty_output><jupyter_text>The `Layer` class takes care of tracking the weights assigned to it as attributes:<jupyter_code># Weights are automatically tracked under the `weights` property.
assert linear_layer.weights == [linear_layer.w, linear_layer.b]<jupyter_output><empty_output><jupyter_text>Note that's also a shortcut method for creating weights: `add_weight`. Instead of doing```pythonw_init = tf.random_normal_initializer()self.w = tf.Variable(initial_value=w_init(shape=shape, dtype='float32'))```You would typically do:```pythonself.w = self.add_weight(shape=shape, initializer='random_normal')``` It’s good practice to create weights in a separate `build` method, called lazily with the shape of the first inputs seen by your layer. Here, this pattern prevents us from having to specify input_dim in the constructor:<jupyter_code>class Linear(Layer):
  """y = w.x + b"""

  def __init__(self, units=32):
      super(Linear, self).__init__()
      self.units = units

  def build(self, input_shape):
      self.w = self.add_weight(shape=(input_shape[-1], self.units),
                               initializer='random_normal',
                               trainable=True)
      self.b = self.add_weight(shape=(self.units,),
                               initializer='random_normal',
                               trainable=True)

  def call(self, inputs):
      return tf.matmul(inputs, self.w) + self.b


# Instantiate our lazy layer.
linear_layer = Linear(4)

# This will also call `build(input_shape)` and create the weights.
y = linear_layer(tf.ones((2, 2)))
assert len(linear_layer.weights) == 2<jupyter_output><empty_output><jupyter_text>Trainable and non-trainable weights Weights created by layers can be either trainable or non-trainable. They're exposed in `trainable_weights` and `non_trainable_weights`. Here's a layer with a non-trainable weight:<jupyter_code>from tensorflow.keras.layers import Layer

class ComputeSum(Layer):
  """Returns the sum of the inputs."""

  def __init__(self, input_dim):
      super(ComputeSum, self).__init__()
      # Create a non-trainable weight.
      self.total = tf.Variable(initial_value=tf.zeros((input_dim,)),
                               trainable=False)

  def call(self, inputs):
      self.total.assign_add(tf.reduce_sum(inputs, axis=0))
      return self.total  

my_sum = ComputeSum(2)
x = tf.ones((2, 2))

y = my_sum(x)
print(y.numpy())  # [2. 2.]

y = my_sum(x)
print(y.numpy())  # [4. 4.]

assert my_sum.weights == [my_sum.total]
assert my_sum.non_trainable_weights == [my_sum.total]
assert my_sum.trainable_weights == []<jupyter_output>[2. 2.]
[4. 4.]<jupyter_text>Recursively composing layers Layers can be recursively nested to create bigger computation blocks. Each layer will track the weights of its sublayers (both trainable and non-trainable).<jupyter_code># Let's reuse the Linear class
# with a `build` method that we defined above.

class MLP(Layer):
    """Simple stack of Linear layers."""

    def __init__(self):
        super(MLP, self).__init__()
        self.linear_1 = Linear(32)
        self.linear_2 = Linear(32)
        self.linear_3 = Linear(10)

    def call(self, inputs):
        x = self.linear_1(inputs)
        x = tf.nn.relu(x)
        x = self.linear_2(x)
        x = tf.nn.relu(x)
        return self.linear_3(x)

mlp = MLP()

# The first call to the `mlp` object will create the weights.
y = mlp(tf.ones(shape=(3, 64)))

# Weights are recursively tracked.
assert len(mlp.weights) == 6<jupyter_output><empty_output><jupyter_text>Built-in layersKeras provides you with a [wide range of built-in layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers/), so that you don't have to implement your own layers all the time.- Convolution layers- Transposed convolutions- Separateable convolutions- Average and max pooling- Global average and max pooling- LSTM, GRU (with built-in cuDNN acceleration)- BatchNormalization- Dropout- Attention- ConvLSTM2D- etc. Keras follows the principles of exposing good default configurations, so that layers will work fine out of the box for most use cases if you leave keyword arguments to their default value. For instance, the `LSTM` layer uses an orthogonal recurrent matrix initializer by default, and initializes the forget gate bias to one by default.  The `training` argument in `call` Some layers, in particular the `BatchNormalization` layer and the `Dropout` layer, have different behaviors during training and inference. For such layers, it is standard practice to expose a `training` (boolean) argument in the `call` method.By exposing this argument in `call`, you enable the built-in training and evaluation loops (e.g. `fit`) to correctly use the layer in training and inference.<jupyter_code>from tensorflow.keras.layers import Layer

class Dropout(Layer):
  
  def __init__(self, rate):
    super(Dropout, self).__init__()
    self.rate = rate

  def call(self, inputs, training=None):
    if training:
      return tf.nn.dropout(inputs, rate=self.rate)
    return inputs

class MLPWithDropout(Layer):

  def __init__(self):
      super(MLPWithDropout, self).__init__()
      self.linear_1 = Linear(32)
      self.dropout = Dropout(0.5)
      self.linear_3 = Linear(10)

  def call(self, inputs, training=None):
      x = self.linear_1(inputs)
      x = tf.nn.relu(x)
      x = self.dropout(x, training=training)
      return self.linear_3(x)
    
mlp = MLPWithDropout()
y_train = mlp(tf.ones((2, 2)), training=True)
y_test = mlp(tf.ones((2, 2)), training=False)<jupyter_output><empty_output><jupyter_text>A more Functional way of defining models To build deep learning models, you don't have to use object-oriented programming all the time. Layers can also be composed functionally, like this (we call it the "Functional API"):<jupyter_code># We use an `Input` object to describe the shape and dtype of the inputs.
# This is the deep learning equivalent of *declaring a type*.
# The shape argument is per-sample; it does not include the batch size.
# The functional API focused on defining per-sample transformations.
# The model we create will automatically batch the per-sample transformations,
# so that it can be called on batches of data.
inputs = tf.keras.Input(shape=(16,))

# We call layers on these "type" objects
# and they return updated types (new shapes/dtypes).
x = Linear(32)(inputs) # We are reusing the Linear layer we defined earlier.
x = Dropout(0.5)(x) # We are reusing the Dropout layer we defined earlier.
outputs = Linear(10)(x)

# A functional `Model` can be defined by specifying inputs and outputs.
# A model is itself a layer like any other.
model = tf.keras.Model(inputs, outputs)

# A functional model already has weights, before being called on any data.
# That's because we defined its input shape in advance (in `Input`).
assert len(model.weights) == 4

# Let's call our model on some data.
y = model(tf.ones((2, 16)))
assert y.shape == (2, 10)<jupyter_output><empty_output><jupyter_text>The Functional API tends to be more concise than subclassing, and provides a few other advantages (generally the same advantages that functional, typed languages provide over untyped OO development). However, it can only be used to define DAGs of layers -- recursive networks should be defined as `Layer` subclasses instead.Key differences between models defined via subclassing and Functional models are explained in [this blog post](https://medium.com/tensorflow/what-are-symbolic-and-imperative-apis-in-tensorflow-2-0-dfccecb01021).Learn more about the Functional API [here](https://www.tensorflow.org/alpha/guide/keras/functional).In your research workflows, you may often find yourself mix-and-matching OO models and Functional models. For models that are simple stacks of layers with a single input and a single output, you can also use the `Sequential` class which turns a list of layers into a `Model`:<jupyter_code>from tensorflow.keras import Sequential

model = Sequential([Linear(32), Dropout(0.5), Linear(10)])

y = model(tf.ones((2, 16)))
assert y.shape == (2, 10)<jupyter_output><empty_output><jupyter_text>Loss classesKeras features a wide range of built-in loss classes, like `BinaryCrossentropy`, `CategoricalCrossentropy`, `KLDivergence`, etc. They work like this:<jupyter_code>bce = tf.keras.losses.BinaryCrossentropy()
y_true = [0., 0., 1., 1.]  # Targets
y_pred = [1., 1., 1., 0.]  # Predictions
loss = bce(y_true, y_pred)
print('Loss:', loss.numpy())<jupyter_output>Loss: 11.522857<jupyter_text>Note that loss classes are stateless: the output of `__call__` is only a function of the input.  Metric classesKeras also features a wide range of built-in metric classes, such as `BinaryAccuracy`, `AUC`, `FalsePositives`, etc.Unlike losses, metrics are stateful. You update their state using the `update_state` method, and you query the scalar metric result using `result`:<jupyter_code>m = tf.keras.metrics.AUC()
m.update_state([0, 1, 1, 1], [0, 1, 0, 0])
print('Intermediate result:', m.result().numpy())

m.update_state([1, 1, 1, 1], [0, 1, 1, 0])
print('Final result:', m.result().numpy())<jupyter_output>Intermediate result:  0.6666667
Final result:  0.71428573<jupyter_text>The internal state can be cleared with `metric.reset_states`. You can easily roll out your own metrics by subclassing the `Metric` class:- Create the state variables in `__init__`- Update the variables given `y_true` and `y_pred` in `update_state`- Return the metric result in `result`- Clear the state in `reset_states`Here's a quick implementation of a `BinaryTruePositives` metric as a demonstration:<jupyter_code>class BinaryTruePositives(tf.keras.metrics.Metric):

  def __init__(self, name='binary_true_positives', **kwargs):
    super(BinaryTruePositives, self).__init__(name=name, **kwargs)
    self.true_positives = self.add_weight(name='tp', initializer='zeros')

  def update_state(self, y_true, y_pred, sample_weight=None):
    y_true = tf.cast(y_true, tf.bool)
    y_pred = tf.cast(y_pred, tf.bool)

    values = tf.logical_and(tf.equal(y_true, True), tf.equal(y_pred, True))
    values = tf.cast(values, self.dtype)
    if sample_weight is not None:
      sample_weight = tf.cast(sample_weight, self.dtype)
      values = tf.multiply(values, sample_weight)
    self.true_positives.assign_add(tf.reduce_sum(values))

  def result(self):
    return self.true_positives

  def reset_states(self):
    self.true_positive.assign(0)

m = BinaryTruePositives()
m.update_state([0, 1, 1, 1], [0, 1, 0, 0])
print('Intermediate result:', m.result().numpy())

m.update_state([1, 1, 1, 1], [0, 1, 1, 0])
print('Final result:', m.result().numpy())<jupyter_output>Intermediate result: 1.0
Final result: 3.0<jupyter_text>Optimizer classes & a quick end-to-end training loopYou don't normally have to define by hand how to update your variables during gradient descent, like we did in our initial linear regression example. You would usually use one of the built-in Keras optimizer, like `SGD`, `RMSprop`, or `Adam`.Here's a simple MNSIT example that brings together loss classes, metric classes, and optimizers.<jupyter_code>from tensorflow.keras import layers

# Prepare a dataset.
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train[:].reshape(60000, 784).astype('float32') / 255
dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset.shuffle(buffer_size=1024).batch(64)

# Instantiate a simple classification model
model = tf.keras.Sequential([
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(10)
])

# Instantiate a logistic loss function that expects integer targets.
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# Instantiate an accuracy metric.
accuracy = tf.keras.metrics.SparseCategoricalAccuracy()

# Instantiate an optimizer.
optimizer = tf.keras.optimizers.Adam()

# Iterate over the batches of the dataset.
for step, (x, y) in enumerate(dataset):
  # Open a GradientTape.
  with tf.GradientTape() as tape:

    # Forward pass.
    logits = model(x)

    # Loss value for this batch.
    loss_value = loss(y, logits)
     
  # Get gradients of loss wrt the weights.
  gradients = tape.gradient(loss_value, model.trainable_weights)
  
  # Update the weights of our linear layer.
  optimizer.apply_gradients(zip(gradients, model.trainable_weights))

  # Update the running accuracy.
  accuracy.update_state(y, logits)
  
  # Logging.
  if step % 100 == 0:
    print('Step:', step)
    print('Loss from last step: %.3f' % loss_value)
    print('Total running accuracy so far: %.3f' % accuracy.result())<jupyter_output>Step: 0
Loss from last step: 2.330
Total running accuracy so far: 0.047
Step: 100
Loss from last step: 0.183
Total running accuracy so far: 0.828
Step: 200
Loss from last step: 0.228
Total running accuracy so far: 0.873
Step: 300
Loss from last step: 0.175
Total running accuracy so far: 0.893
Step: 400
Loss from last step: 0.164
Total running accuracy so far: 0.905
Step: 500
Loss from last step: 0.234
Total running accuracy so far: 0.914
Step: 600
Loss from last step: 0.231
Total running accuracy so far: 0.921
Step: 700
Loss from last step: 0.149
Total running accuracy so far: 0.926
Step: 800
Loss from last step: 0.268
Total running accuracy so far: 0.930
Step: 900
Loss from last step: 0.061
Total running accuracy so far: 0.933<jupyter_text>We can reuse our `SparseCategoricalAccuracy` metric instance to implement a testing loop:<jupyter_code>x_test = x_test[:].reshape(10000, 784).astype('float32') / 255
test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
test_dataset = test_dataset.batch(128)

accuracy.reset_states()  # This clears the internal state of the metric

for step, (x, y) in enumerate(test_dataset):
  logits = model(x)
  accuracy.update_state(y, logits)

print('Final test accuracy: %.3f' % accuracy.result())<jupyter_output>Final test accuracy: 0.963<jupyter_text>The `add_loss` methodSometimes you need to compute loss values on the fly during a foward pass (especially regularization losses). Keras allows you to compute loss values at any time, and to recursively keep track of them via the `add_loss` method.Here's an example of a layer that adds a sparsity regularization loss based on the L2 norm of the inputs:<jupyter_code>from tensorflow.keras.layers import Layer

class ActivityRegularization(Layer):
  """Layer that creates an activity sparsity regularization loss."""
  
  def __init__(self, rate=1e-2):
    super(ActivityRegularization, self).__init__()
    self.rate = rate
  
  def call(self, inputs):
    # We use `add_loss` to create a regularization loss
    # that depends on the inputs.
    self.add_loss(self.rate * tf.reduce_sum(tf.square(inputs)))
    return inputs<jupyter_output><empty_output><jupyter_text>Loss values added via `add_loss` can be retrieved in the `.losses` list property of any `Layer` or `Model`:<jupyter_code>from tensorflow.keras import layers

class SparseMLP(Layer):
  """Stack of Linear layers with a sparsity regularization loss."""

  def __init__(self, output_dim):
      super(SparseMLP, self).__init__()
      self.dense_1 = layers.Dense(32, activation=tf.nn.relu)
      self.regularization = ActivityRegularization(1e-2)
      self.dense_2 = layers.Dense(output_dim)

  def call(self, inputs):
      x = self.dense_1(inputs)
      x = self.regularization(x)
      return self.dense_2(x)
    

mlp = SparseMLP(1)
y = mlp(tf.ones((10, 10)))

print(mlp.losses)  # List containing one float32 scalar<jupyter_output>[<tf.Tensor: id=201583, shape=(), dtype=float32, numpy=1.0274899>]<jupyter_text>These losses are cleared by the top-level layer at the start of each forward pass -- they don't accumulate. So `layer.losses` always contain only the losses created during the last forward pass. You would typically use these losses by summing them before computing your gradients when writing a training loop.<jupyter_code># Losses correspond to the *last* forward pass.
mlp = SparseMLP(1)
mlp(tf.ones((10, 10)))
assert len(mlp.losses) == 1
mlp(tf.ones((10, 10)))
assert len(mlp.losses) == 1  # No accumulation.

# Let's demonstrate how to use these losses in a training loop.

# Prepare a dataset.
(x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
dataset = tf.data.Dataset.from_tensor_slices(
    (x_train.reshape(60000, 784).astype('float32') / 255, y_train))
dataset = dataset.shuffle(buffer_size=1024).batch(64)

# A new MLP.
mlp = SparseMLP(10)

# Loss and optimizer.
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)

for step, (x, y) in enumerate(dataset):
  with tf.GradientTape() as tape:
    # Forward pass.
    logits = mlp(x)

    # External loss value for this batch.
    loss = loss_fn(y, logits)
    
    # Add the losses created during the forward pass.
    loss += sum(mlp.losses)
     
    # Get gradients of loss wrt the weights.
    gradients = tape.gradient(loss, mlp.trainable_weights)
  
  # Update the weights of our linear layer.
  optimizer.apply_gradients(zip(gradients, mlp.trainable_weights))
  
  # Logging.
  if step % 100 == 0:
    print('Loss at step %d: %.3f' % (step, loss))<jupyter_output>Loss at step 0: 4.389
Loss at step 100: 2.301
Loss at step 200: 2.278
Loss at step 300: 2.210
Loss at step 400: 2.157
Loss at step 500: 2.041
Loss at step 600: 1.945
Loss at step 700: 1.932
Loss at step 800: 1.818
Loss at step 900: 2.024<jupyter_text>A detailed end-to-end example: a Variational AutoEncoder (VAE)If you want to take a break from the basics and look at a slightly more advanced example, check out this [Variational AutoEncoder](https://www.tensorflow.org/guide/keras/custom_layers_and_modelsputting_it_all_together_an_end-to-end_example) implementation that demonstrates everything you've learned so far:- Subclassing `Layer`- Recursive layer composition- Loss classes and metric classes- `add_loss`- `GradientTape`  Using built-in training loops It would be a bit silly if you had to write your own low-level training loops every time for simple use cases. Keras provides you with a built-in training loop on the `Model` class. If you want to use it, either subclass from the `Model` class, or create a `Functional` or `Sequential` model.To demonstrate it, let's reuse the MNIST setup from above:<jupyter_code># Prepare a dataset.
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 784).astype('float32') / 255
dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
dataset = dataset.shuffle(buffer_size=1024).batch(64)

# Instantiate a simple classification model
model = tf.keras.Sequential([
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(10)
])

# Instantiate a logistic loss function that expects integer targets.
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# Instantiate an accuracy metric.
accuracy = tf.keras.metrics.SparseCategoricalAccuracy()

# Instantiate an optimizer.
optimizer = tf.keras.optimizers.Adam()<jupyter_output><empty_output><jupyter_text>First, call `compile` to configure the optimizer, loss, and metrics to monitor.<jupyter_code>model.compile(optimizer=optimizer, loss=loss, metrics=[accuracy])<jupyter_output><empty_output><jupyter_text>Then we call `fit` on our model to pass it the data:<jupyter_code>model.fit(dataset, epochs=3)<jupyter_output>Epoch 1/3
938/938 [==============================] - 9s 10ms/step - loss: 0.2160 - sparse_categorical_accuracy: 0.9370
Epoch 2/3
938/938 [==============================] - 6s 6ms/step - loss: 0.0831 - sparse_categorical_accuracy: 0.9745
Epoch 3/3
938/938 [==============================] - 6s 6ms/step - loss: 0.0571 - sparse_categorical_accuracy: 0.9817<jupyter_text>Done!**Note:** When you use `fit`, by default execution uses static graphs, so you don't need to add any `tf.function` decorators to your model or your layers.Now let's test it:<jupyter_code>x_test = x_test[:].reshape(10000, 784).astype('float32') / 255
test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))
test_dataset = test_dataset.batch(128)

loss, acc = model.evaluate(test_dataset)
print('loss: %.3f - acc: %.3f' % (loss, acc))<jupyter_output>79/79 [==============================] - 0s 5ms/step - loss: 0.0853 - sparse_categorical_accuracy: 0.9756
loss: 0.085 - acc: 0.976<jupyter_text>Note that you can also monitor your loss and metrics on some validation data during `fit`.Also, you can call `fit` directly on Numpy arrays, so no need for the dataset conversion:<jupyter_code>(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 784).astype('float32') / 255

num_val_samples = 10000
x_val = x_train[-num_val_samples:]
y_val = y_train[-num_val_samples:]
x_train = x_train[:-num_val_samples]
y_train = y_train[:-num_val_samples]

# Instantiate a simple classification model
model = tf.keras.Sequential([
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(10)
])

# Instantiate a logistic loss function that expects integer targets.
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# Instantiate an accuracy metric.
accuracy = tf.keras.metrics.SparseCategoricalAccuracy()

# Instantiate an optimizer.
optimizer = tf.keras.optimizers.Adam()

model.compile(optimizer=optimizer,
              loss=loss,
              metrics=[accuracy])
model.fit(x_train, y_train,
          validation_data=(x_val, y_val),
          epochs=3,
          batch_size=64)<jupyter_output>Train on 50000 samples, validate on 10000 samples
Epoch 1/3
50000/50000 [==============================] - 5s 104us/sample - loss: 0.2447 - sparse_categorical_accuracy: 0.9283 - val_loss: 0.1155 - val_sparse_categorical_accuracy: 0.9663
Epoch 2/3
50000/50000 [==============================] - 5s 94us/sample - loss: 0.0947 - sparse_categorical_accuracy: 0.9713 - val_loss: 0.1011 - val_sparse_categorical_accuracy: 0.9703
Epoch 3/3
50000/50000 [==============================] - 5s 91us/sample - loss: 0.0620 - sparse_categorical_accuracy: 0.9803 - val_loss: 0.0803 - val_sparse_categorical_accuracy: 0.9773<jupyter_text>CallbacksOne of the neat features of `fit` (besides built-in support for sample weighting and class weighting) is that you can easily customize what happens during training and evaluation by using [callbacks](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/).A callback is an object that is called at different points during training (e.g. at the end of every batch or at the end of every epoch) and takes actions, such as saving a model, mutating variables on the model, loading a checkpoint, stopping training, etc.There's a bunch of built-in callback available, like `ModelCheckpoint` to save your models after each epoch during training, or `EarlyStopping`, which interrupts training when your validation metrics start stalling.And you can easily [write your own callbacks](https://www.tensorflow.org/guide/keras/custom_callback).<jupyter_code># Instantiate a simple classification model
model = tf.keras.Sequential([
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(256, activation=tf.nn.relu),
  layers.Dense(10)
])

# Instantiate a logistic loss function that expects integer targets.
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

# Instantiate an accuracy metric.
accuracy = tf.keras.metrics.SparseCategoricalAccuracy()

# Instantiate an optimizer.
optimizer = tf.keras.optimizers.Adam()

model.compile(optimizer=optimizer,
              loss=loss,
              metrics=[accuracy])

# Instantiate some callbacks
callbacks = [tf.keras.callbacks.EarlyStopping(),
             tf.keras.callbacks.ModelCheckpoint(filepath='my_model.keras',
                                                save_best_only=True)]

model.fit(x_train, y_train,
          validation_data=(x_val, y_val),
          epochs=30,
          batch_size=64,
          callbacks=callbacks)<jupyter_output>Train on 50000 samples, validate on 10000 samples
Epoch 1/30
50000/50000 [==============================] - 6s 113us/sample - loss: 0.2405 - sparse_categorical_accuracy: 0.9280 - val_loss: 0.1103 - val_sparse_categorical_accuracy: 0.9669
Epoch 2/30
50000/50000 [==============================] - 5s 91us/sample - loss: 0.0916 - sparse_categorical_accuracy: 0.9716 - val_loss: 0.0835 - val_sparse_categorical_accuracy: 0.9745
Epoch 3/30
50000/50000 [==============================] - 5s 107us/sample - loss: 0.0617 - sparse_categorical_accuracy: 0.9804 - val_loss: 0.0896 - val_sparse_categorical_accuracy: 0.9738
{'fraction_non_alphanumeric': 0.11530293894030359, 'fraction_numerical': 0.033374406837427574, 'mean_word_length': 4.765417170495768, 'pattern_counts': {'":': 0, '<': 153, '<?xml version=': 0, '>': 154, 'https://': 13, 'lorem ipsum': 0, 'www.': 11, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '43', 'dir': 'jupyter-structured-clean-dedup', 'id': '35071', 'license': 'Apache-2.0', 'n_tokens_mistral': 13309, 'n_tokens_neox': 12147, 'n_words': 3941, 'path': 'deep-learning/Tensorflow-2.x/Examples/TensorFlow_2_0_+_Keras_Crash_Course.ipynb', 'repo_name': 'AadityaGupta/Artificial-Intelligence-Deep-Learning-Machine-Learning-Tutorials', 'size': '38143'}