<jupyter_start><jupyter_text>The following additional libraries are needed to run thisnotebook. Note that running on Colab is experimental, please report a Githubissue if you have any problem.<jupyter_code>!pip install d2l==0.17.2<jupyter_output>Requirement already satisfied: d2l==0.17.2 in /usr/local/lib/python3.7/dist-packages (0.17.2)
Requirement already satisfied: numpy==1.18.5 in /usr/local/lib/python3.7/dist-packages (from d2l==0.17.2) (1.18.5)
Requirement already satisfied: jupyter==1.0.0 in /usr/local/lib/python3.7/dist-packages (from d2l==0.17.2) (1.0.0)
Requirement already satisfied: requests==2.25.1 in /usr/local/lib/python3.7/dist-packages (from d2l==0.17.2) (2.25.1)
Requirement already satisfied: matplotlib==3.3.3 in /usr/local/lib/python3.7/dist-packages (from d2l==0.17.2) (3.3.3)
Requirement already satisfied: pandas==1.2.2 in /usr/local/lib/python3.7/dist-packages (from d2l==0.17.2) (1.2.2)
Requirement already satisfied: qtconsole in /usr/local/lib/python3.7/dist-packages (from jupyter==1.0.0->d2l==0.17.2) (5.2.2)
Requirement already satisfied: notebook in /usr/local/lib/python3.7/dist-packages (from jupyter==1.0.0->d2l==0.17.2) (5.3.1)
Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-p[...]<jupyter_text>Batch Normalization:label:`sec_batch_norm`Training deep neural networks is difficult.And getting them to converge in a reasonable amount of time can be tricky.In this section, we describe *batch normalization*, a popular and effective techniquethat consistently accelerates the convergence of deep networks :cite:`Ioffe.Szegedy.2015`.Together with residual blocks---covered later in :numref:`sec_resnet`---batch normalizationhas made it possible for practitionersto routinely train networks with over 100 layers. Training Deep NetworksTo motivate batch normalization, let us reviewa few practical challenges that arisewhen training machine learning models and neural networks in particular.First, choices regarding data preprocessing often make an enormous difference in the final results.Recall our application of MLPs to predicting house prices (:numref:`sec_kaggle_house`).Our first step when working with real datawas to standardize our input featuresto each have a mean of zero and variance of one.Intuitively, this standardization plays nicely with our optimizersbecause it puts the parameters *a priori* at a similar scale.   Second, for a typical MLP or CNN, as we train,the variables (e.g., affine transformation outputs in MLP)in intermediate layers may take values with widely varying magnitudes:both along the layers from the input to the output, across units in the same layer,and over time due to our updates to the model parameters.The inventors of batch normalization postulated informallythat this drift in the distribution of such variables could hamper the convergence of the network.Intuitively, we might conjecture that if onelayer has variable values that are 100 times that of another layer,this might necessitate compensatory adjustments in the learning rates.   Third, deeper networks are complex and easily capable of overfitting.This means that regularization becomes more critical.Batch normalization is applied to individual layers(optionally, to all of them) and works as follows:In each training iteration,we first normalize the inputs (of batch normalization)by subtracting their mean anddividing by their standard deviation,where both are estimated based on the statistics of the current minibatch.Next, we apply a scale coefficient and a scale offset.It is precisely due to this *normalization* based on *batch* statisticsthat *batch normalization* derives its name.Note that if we tried to apply batch normalization with minibatches of size 1,we would not be able to learn anything.That is because after subtracting the means,each hidden unit would take value 0!As you might guess, since we are devoting a whole section to batch normalization,with large enough minibatches, the approach proves effective and stable.One takeaway here is that when applying batch normalization,the choice of batch size may beeven more significant than without batch normalization.Formally, denoting by $\mathbf{x} \in \mathcal{B}$ an input to batch normalization ($\mathrm{BN}$)that is from a minibatch $\mathcal{B}$,batch normalization transforms $\mathbf{x}$according to the following expression:$$\mathrm{BN}(\mathbf{x}) = \boldsymbol{\gamma} \odot \frac{\mathbf{x} - \hat{\boldsymbol{\mu}}_\mathcal{B}}{\hat{\boldsymbol{\sigma}}_\mathcal{B}} + \boldsymbol{\beta}.$$:eqlabel:`eq_batchnorm`In :eqref:`eq_batchnorm`,$\hat{\boldsymbol{\mu}}_\mathcal{B}$ is the  sample meanand $\hat{\boldsymbol{\sigma}}_\mathcal{B}$ is the sample standard deviation of the minibatch $\mathcal{B}$.After applying standardization,the resulting minibatchhas zero mean and unit variance.Because the choice of unit variance(vs. some other magic number) is an arbitrary choice,we commonly include elementwise*scale parameter* $\boldsymbol{\gamma}$ and *shift parameter* $\boldsymbol{\beta}$that have the same shape as $\mathbf{x}$.Note that $\boldsymbol{\gamma}$ and $\boldsymbol{\beta}$ are parameters that need to be learned jointly with the other model parameters.Consequently, the variable magnitudesfor intermediate layers cannot diverge during trainingbecause batch normalization actively centers and rescales them backto a given mean and size (via $\hat{\boldsymbol{\mu}}_\mathcal{B}$ and ${\hat{\boldsymbol{\sigma}}_\mathcal{B}}$).One piece of practitioner's intuition or wisdomis that batch normalization seems to allow for more aggressive learning rates.Formally, we calculate $\hat{\boldsymbol{\mu}}_\mathcal{B}$ and ${\hat{\boldsymbol{\sigma}}_\mathcal{B}}$ in :eqref:`eq_batchnorm` as follows:$$\begin{aligned} \hat{\boldsymbol{\mu}}_\mathcal{B} &= \frac{1}{|\mathcal{B}|} \sum_{\mathbf{x} \in \mathcal{B}} \mathbf{x},\\\hat{\boldsymbol{\sigma}}_\mathcal{B}^2 &= \frac{1}{|\mathcal{B}|} \sum_{\mathbf{x} \in \mathcal{B}} (\mathbf{x} - \hat{\boldsymbol{\mu}}_{\mathcal{B}})^2 + \epsilon.\end{aligned}$$Note that we add a small constant $\epsilon > 0$to the variance estimateto ensure that we never attempt division by zero,even in cases where the empirical variance estimate might vanish.The estimates $\hat{\boldsymbol{\mu}}_\mathcal{B}$ and ${\hat{\boldsymbol{\sigma}}_\mathcal{B}}$ counteract the scaling issueby using noisy estimates of mean and variance.You might think that this noisiness should be a problem.As it turns out, this is actually beneficial.This turns out to be a recurring theme in deep learning.For reasons that are not yet well-characterized theoretically,various sources of noise in optimizationoften lead to faster training and less overfitting:this variation appears to act as a form of regularization.In some preliminary research,:cite:`Teye.Azizpour.Smith.2018` and :cite:`Luo.Wang.Shao.ea.2018`relate the properties of batch normalization to Bayesian priors and penalties respectively.In particular, this sheds some light on the puzzleof why batch normalization works best for moderate minibatches sizes in the $50 \sim 100$ range.Fixing a trained model, you might thinkthat we would prefer using the entire datasetto estimate the mean and variance.Once training is complete, why would we wantthe same image to be classified differently,depending on the batch in which it happens to reside?During training, such exact calculation is infeasiblebecause the intermediate variablesfor all data exampleschange every time we update our model.However, once the model is trained,we can calculate the means and variancesof each layer's variables based on the entire dataset.Indeed this is standard practice formodels employing batch normalizationand thus batch normalization layers function differentlyin *training mode* (normalizing by minibatch statistics)and in *prediction mode* (normalizing by dataset statistics).We are now ready to take a look at how batch normalization works in practice. Batch Normalization LayersBatch normalization implementations for fully-connected layersand convolutional layers are slightly different.We discuss both cases below.Recall that one key differences between batch normalization and other layersis that because batch normalization operates on a full minibatch at a time,we cannot just ignore the batch dimensionas we did before when introducing other layers. Fully-Connected LayersWhen applying batch normalization to fully-connected layers,the original paper inserts batch normalization after the affine transformationand before the nonlinear activation function (later applications may insert batch normalization right after activation functions) :cite:`Ioffe.Szegedy.2015`.Denoting the input to the fully-connected layer by $\mathbf{x}$,the affine transformationby $\mathbf{W}\mathbf{x} + \mathbf{b}$ (with the weight parameter $\mathbf{W}$ and the bias parameter $\mathbf{b}$),and the activation function by $\phi$,we can express the computation of a batch-normalization-enabled,fully-connected layer output $\mathbf{h}$ as follows:$$\mathbf{h} = \phi(\mathrm{BN}(\mathbf{W}\mathbf{x} + \mathbf{b}) ).$$Recall that mean and variance are computedon the *same* minibatch on which the transformation is applied. Convolutional LayersSimilarly, with convolutional layers,we can apply batch normalization after the convolutionand before the nonlinear activation function.When the convolution has multiple output channels,we need to carry out batch normalizationfor *each* of the outputs of these channels,and each channel has its own scale and shift parameters,both of which are scalars.Assume that our minibatches contain $m$ examplesand that for each channel,the output of the convolution has height $p$ and width $q$.For convolutional layers, we carry out each batch normalizationover the $m \cdot p \cdot q$ elements per output channel simultaneously.Thus, we collect the values over all spatial locationswhen computing the mean and varianceand consequently apply the same mean and variancewithin a given channelto normalize the value at each spatial location. Batch Normalization During PredictionAs we mentioned earlier, batch normalization typically behaves differentlyin training mode and prediction mode.First, the noise in the sample mean and the sample variancearising from estimating each on minibatchesare no longer desirable once we have trained the model.Second, we might not have the luxuryof computing per-batch normalization statistics.For example,we might need to apply our model to make one prediction at a time.Typically, after training, we use the entire datasetto compute stable estimates of the variable statisticsand then fix them at prediction time.Consequently, batch normalization behaves differently during training and at test time.Recall that dropout also exhibits this characteristic. (**Implementation from Scratch**)Below, we implement a batch normalization layer with tensors from scratch.<jupyter_code>import tensorflow as tf
from d2l import tensorflow as d2l


def batch_norm(X, gamma, beta, moving_mean, moving_var, eps):
    # Compute reciprocal of square root of the moving variance elementwise
    inv = tf.cast(tf.math.rsqrt(moving_var + eps), X.dtype)
    # Scale and shift
    inv *= gamma
    Y = X * inv + (beta - moving_mean * inv)
    return Y<jupyter_output><empty_output><jupyter_text>We can now [**create a proper `BatchNorm` layer.**]Our layer will maintain proper parametersfor scale `gamma` and shift `beta`,both of which will be updated in the course of training.Additionally, our layer will maintainmoving averages of the means and variancesfor subsequent use during model prediction.Putting aside the algorithmic details,note the design pattern underlying our implementation of the layer.Typically, we define the mathematics in a separate function, say `batch_norm`.We then integrate this functionality into a custom layer,whose code mostly addresses bookkeeping matters,such as moving data to the right device context,allocating and initializing any required variables,keeping track of moving averages (here for mean and variance), and so on.This pattern enables a clean separation of mathematics from boilerplate code.Also note that for the sake of conveniencewe did not worry about automatically inferring the input shape here,thus we need to specify the number of features throughout.Do not worry, the high-level batch normalization APIs in the deep learning framework will care of this for us and we will demonstrate that later.<jupyter_code>class BatchNorm(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super(BatchNorm, self).__init__(**kwargs)

    def build(self, input_shape):
        weight_shape = [input_shape[-1], ]
        # The scale parameter and the shift parameter (model parameters) are
        # initialized to 1 and 0, respectively
        self.gamma = self.add_weight(name='gamma', shape=weight_shape,
            initializer=tf.initializers.ones, trainable=True)
        self.beta = self.add_weight(name='beta', shape=weight_shape,
            initializer=tf.initializers.zeros, trainable=True)
        # The variables that are not model parameters are initialized to 0
        self.moving_mean = self.add_weight(name='moving_mean',
            shape=weight_shape, initializer=tf.initializers.zeros,
            trainable=False)
        self.moving_variance = self.add_weight(name='moving_variance',
            shape=weight_shape, initializer=tf.initializers.ones,
            trainable=False)
        super(BatchNorm, self).build(input_shape)

    def assign_moving_average(self, variable, value):
        momentum = 0.9
        delta = variable * momentum + value * (1 - momentum)
        return variable.assign(delta)

    @tf.function
    def call(self, inputs, training):
        if training:
            axes = list(range(len(inputs.shape) - 1))
            batch_mean = tf.reduce_mean(inputs, axes, keepdims=True)
            batch_variance = tf.reduce_mean(tf.math.squared_difference(
                inputs, tf.stop_gradient(batch_mean)), axes, keepdims=True)
            batch_mean = tf.squeeze(batch_mean, axes)
            batch_variance = tf.squeeze(batch_variance, axes)
            mean_update = self.assign_moving_average(
                self.moving_mean, batch_mean)
            variance_update = self.assign_moving_average(
                self.moving_variance, batch_variance)
            self.add_update(mean_update)
            self.add_update(variance_update)
            mean, variance = batch_mean, batch_variance
        else:
            mean, variance = self.moving_mean, self.moving_variance
        output = batch_norm(inputs, moving_mean=mean, moving_var=variance,
            beta=self.beta, gamma=self.gamma, eps=1e-5)
        return output<jupyter_output><empty_output><jupyter_text>[**Applying Batch Normalization in LeNet**]To see how to apply `BatchNorm` in context,below we apply it to a traditional LeNet model (:numref:`sec_lenet`).Recall that batch normalization is appliedafter the convolutional layers or fully-connected layersbut before the corresponding activation functions.<jupyter_code># Recall that this has to be a function that will be passed to `d2l.train_ch6`
# so that model building or compiling need to be within `strategy.scope()` in
# order to utilize the CPU/GPU devices that we have
def net():
    return tf.keras.models.Sequential([
        tf.keras.layers.Conv2D(filters=6, kernel_size=5,
                               input_shape=(28, 28, 1)),
        BatchNorm(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
        tf.keras.layers.Conv2D(filters=16, kernel_size=5),
        BatchNorm(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(120),
        BatchNorm(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.Dense(84),
        BatchNorm(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.Dense(10)]
    )<jupyter_output><empty_output><jupyter_text>As before, we will [**train our network on the Fashion-MNIST dataset**].This code is virtually identical to that when we first trained LeNet (:numref:`sec_lenet`).The main difference is the larger learning rate.<jupyter_code>lr, num_epochs, batch_size = 1.0, 10, 256
train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)
net = d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())<jupyter_output>loss 0.271, train acc 0.900, test acc 0.860
20630.2 examples/sec on /GPU:0<jupyter_text>Let us [**have a look at the scale parameter `gamma`and the shift parameter `beta`**] learnedfrom the first batch normalization layer.<jupyter_code>tf.reshape(net.layers[1].gamma, (-1,)), tf.reshape(net.layers[1].beta, (-1,))<jupyter_output><empty_output><jupyter_text>[**Concise Implementation**]Compared with the `BatchNorm` class,which we just defined ourselves,we can use the `BatchNorm` class defined in high-level APIs from the deep learning framework directly.The code looks virtually identicalto our implementation above.<jupyter_code>def net():
    return tf.keras.models.Sequential([
        tf.keras.layers.Conv2D(filters=6, kernel_size=5,
                               input_shape=(28, 28, 1)),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
        tf.keras.layers.Conv2D(filters=16, kernel_size=5),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.AvgPool2D(pool_size=2, strides=2),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(120),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.Dense(84),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Activation('sigmoid'),
        tf.keras.layers.Dense(10),
    ])<jupyter_output><empty_output><jupyter_text>Below, we [**use the same hyperparameters to train our model.**]Note that as usual, the high-level API variant runs much fasterbecause its code has been compiled to C++ or CUDAwhile our custom implementation must be interpreted by Python.<jupyter_code>d2l.train_ch6(net, train_iter, test_iter, num_epochs, lr, d2l.try_gpu())<jupyter_output>loss 0.259, train acc 0.904, test acc 0.848
46071.5 examples/sec on /GPU:0
{'fraction_non_alphanumeric': 0.08613162291812645, 'fraction_numerical': 0.01439120357893602, 'mean_word_length': 4.684436274509804, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '208654', 'license': 'Unlicense', 'n_tokens_mistral': 5140, 'n_tokens_neox': 4817, 'n_words': 1909, 'path': 'Week 12/Batch Normalization.ipynb', 'repo_name': 'Hanif-2610/Machine-Learning-Homework-Project', 'size': '18553'}