<jupyter_start><jupyter_text>Initialization Consider the following network:<jupyter_code>class Network(nn.Module):
    
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(nn.Linear(10, 100, bias=False),
                                    nn.ReLU(),
                                    nn.Linear(100, 100, bias=False),
                                    nn.ReLU(),
                                    nn.Linear(100, 100, bias=False),
                                    nn.ReLU(),
                                    nn.Linear(100, 1, bias=False),
                                   )
    
    def forward(self, x):
        return self.layers(x)

    
def hook_fun(module, input, output):
    activations.append(output)<jupyter_output><empty_output><jupyter_text>Execute the following cells and observe the statistics of the layer outputs:<jupyter_code>x = torch.randn(100, 10)
net = Network()
def hook_fun(module, input, output):   #check for register_forward_hook in pytorch doc
    activations.append(output)
hooks = []
activations = []

for layer in net.layers:
    hook = layer.register_forward_hook(hook_fun) #this is not executed now, it creates a hook and will be executed when forward is called
    hooks.append(hook)

with torch.no_grad():
    res = net(x)

for hook in hooks:
    hook.remove()
activations[0]
plt.figure(figsize=(15, 15))

for i, act in enumerate(activations):
    plt.subplot(4, 2, i+1)
    plt.hist(act.reshape(-1), bins=9, density=True, range=(-2.1,2.1))
    plt.gca().set_ylim(0, 1)

print(res.var())<jupyter_output>tensor(0.0005)<jupyter_text>Kaiming Initialization Implement the initialization method by <NAME> as introduced in the lecture, also see https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf. Therefore complete the ``init_weights`` function in ``utils.py``. For recursively applying a method to the network's modules, we use ``torch.nn.Module.apply``, https://pytorch.org/docs/stable/nn.html?highlight=applytorch.nn.Module.apply.<jupyter_code>net.apply(init_weights);
hooks = []
activations = []

for layer in net.layers:
    hook = layer.register_forward_hook(hook_fun)
    hooks.append(hook)

with torch.no_grad():
    res = net(x)

for hook in hooks:
    hook.remove()
plt.figure(figsize=(15, 15))

for i, act in enumerate(activations):
    plt.subplot(4, 2, i+1)
    plt.hist(act.reshape(-1), bins=9, density=True, range=(-2.1,2.1))
    plt.gca().set_ylim(0, 1)

print(res.var())
print(net.layers[6].weight.var() * net.layers[6].in_features / 2)<jupyter_output>tensor(1.1740, grad_fn=<DivBackward0>)<jupyter_text>Batch Normalization From the lecture you already know a different strategy for stabilizing the training of deep networks: Batch normalization. Consider an input of size $B \times C$. For each channel $C$ the batch norm layer takes all $B$ values of the batch elements and normalizes them accordingly. Afterwards it rescales the output with a learnable deviation and adds a bias parameter per channel, see e.g. https://pytorch.org/docs/stable/nn.html?highlight=applytorch.nn.tchNorm2d and https://arxiv.org/abs/1502.03167. At inference time, the batch norm layer 'normalizes' one input image by using running estimates of the deviations and biases of the inputs so far. Implement ``BatchNorm`` in ``utils.py``. Initialize the learnable parameters in such a way that the layer initially returns the channel-wise normed input. At the first forward pass, set the running mean and variance to those of the first mini-batch. Afterwards update them by taking the weighted sum of the current and the running means and variances. Further details are provided in the comments in ``utils.py``.<jupyter_code>bn = BatchNorm(10)

torch.manual_seed(0)
x1 = torch.randn(4, 10)
x2 = torch.randn(4, 10)
x3 = torch.randn(4, 10)

with torch.no_grad():
    res1 = bn(x1)
    bn.theta_sigma.data = bn.theta_sigma * 2
    bn.theta_mu.data = bn.theta_mu + 1
    res2 = bn(x2)
    bn.eval()
    res3 = bn(x3)

assert (res1.mean(dim=0) - 0).abs().max()<1e-6 and (res1.var(dim=0) - 1).abs().max()<1e-3
assert (res2.mean(dim=0) - 1).abs().max()<1e-6 and (res2.var(dim=0) - 4).abs().max()<1e-3
assert (res3.mean() - 1.3529).abs().max()<1e-3 and (res3.var() - 4.6663).abs().max()<1e-3<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12635462301129813, 'fraction_numerical': 0.0341249711782338, 'mean_word_length': 3.404060913705584, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 17, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '542074', 'license': 'MIT', 'n_tokens_mistral': 1521, 'n_tokens_neox': 1424, 'n_words': 389, 'path': 'DeepLearningCourse-Uni/Exercise_6_Initialisation&BatchNormalisation/Initialization.ipynb', 'repo_name': 'anish-pratheepkumar/AI-Machine-Learning-and-Deep-Learning', 'size': '4341'}