<jupyter_start><jupyter_text>RoadMap 10 - Torch NN Layers - Initializers        1.   torch.nn.init.calculate_gain   - Return the recommended gain value for the given nonlinearity function.    2.   torch.nn.init.uniform_         - Fills the input Tensor with values drawn from the                                            uniform distribution U(a,b).    3.   torch.nn.init.normal_          - Fills the input Tensor with values drawn from the normal                                            distribution N(mean,std).    4.   torch.nn.init.constant_        - Fills the input Tensor with the value val.    5.   torch.nn.init.eye_             - Fills the 2-dimensional input Tensor with the identity matrix.     6.   torch.nn.init.dirac_           - Fills the {3, 4, 5}-dimensional input Tensor with the Dirac                                            delta function.    7.   torch.nn.init.xavier_uniform_  - Fills the input Tensor with values according to the method described                                            in “Understanding the difficulty of training deep feedforward                                            neural networks” - <NAME>. & <NAME>. (2010), using a                                            uniform distribution.    8.   torch.nn.init.xavier_normal_   - Fills the input Tensor with values according to the method described                                            in “Understanding the difficulty of training deep feedforward                                            neural networks” - <NAME>. & <NAME>. (2010), using a                                            normal distribution.     9.   torch.nn.init.kaiming_uniform_ - Fills the input Tensor with values according to the method described                                            in “Delving deep into rectifiers: Surpassing human-level performance                                            on ImageNet classification” - He, K. et al. (2015), using a                                            uniform distribution.     10.  torch.nn.init.kaiming_normal_  - Fills the input Tensor with values according to the method described                                           in “Delving deep into rectifiers: Surpassing human-level performance                                            on ImageNet classification” - He, K. et al. (2015), using a                                            normal distribution.    11.  torch.nn.init.orthogonal_      - Fills the input Tensor with a (semi) orthogonal matrix, as described                                            in “Exact solutions to the nonlinear dynamics of learning in deep                                           linear neural networks” - <NAME>. et al. (2013).     12.  torch.nn.init.sparse_          - Fills the 2D input Tensor as a sparse matrix, where the non-zero                                            elements will be drawn from the normal distribution N(0,0.01),                                            as described in “Deep learning via Hessian-free                                            optimization” - <NAME>. (2010).<jupyter_code>import os
import sys
import torch
import numpy as np
import torch.nn as nn
from torchvision import transforms, datasets
from PIL import Image
import cv2
import matplotlib.pyplot as plt
import torchvision<jupyter_output><empty_output><jupyter_text>Extra Blog Resources1. https://towardsdatascience.com/weight-initialization-techniques-in-neural-networks-26c649eb3b78    2. https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79    3. https://medium.com/usf-msds/deep-learning-best-practices-1-weight-initialization-14e5c0295b94<jupyter_code># Input
image_name = "dog.jpg"
image_pil = Image.open(image_name)

transform = transforms.Compose([transforms.ToTensor()])
image = transform(image_pil).float()
image_nchw = torch.unsqueeze(image, 0)
print(image_nchw.size())
plt.imshow(image_pil)
# Gain from non-linearity

'''
1. nn.init.calculate_gain - Return the recommended gain value for the given nonlinearity function.
        - nonlinearity – the non-linear function (nn.functional name)
        - param – optional parameter for the non-linear function
'''

gain_leaky_relu = nn.init.calculate_gain('leaky_relu')
gain_conv2d = nn.init.calculate_gain('conv2d')

print("Gain of Leaky ReLU - ", gain_leaky_relu)
print("Gain of Convolution 2d - ", gain_conv2d)
# Uniform Filler function

'''
2. nn.init.uniform_ - Fills the input Tensor with values drawn from the uniform distribution U(a,b).
        - tensor – an n-dimensional torch.Tensor
        - a – the lower bound of the uniform distribution
        - b – the upper bound of the uniform distribution
'''

w = torch.empty(3, 5)
w = nn.init.uniform_(w, a=2, b=5)

print("w = ", w)
# Normal filler function

'''
3. nn.init.normal_ - Fills the input Tensor with values drawn from the normal distribution N(mean,std).
        - tensor – an n-dimensional torch.Tensor
        - mean – the mean of the normal distribution
        - std – the standard deviation of the normal distribution
'''

w = torch.empty(3, 5)
w = nn.init.uniform_(w, 0.0, 1.0)

print("w = ", w)
# Constant filler function

'''
4. nn.init.constant_ - Fills the input Tensor with the value val.
        - tensor – an n-dimensional torch.Tensor
        - val – the value to fill the tensor with
'''
w = torch.empty(3, 5)
w = nn.init.constant_(w, 1.5)

print("w = ", w)
# Eye filler function

'''
5. nn.init.eye_ - Fills the 2-dimensional input Tensor with the identity matrix. 
    Preserves the identity of the inputs in Linear layers, where as many inputs are preserved as possible.
'''

w = torch.empty(3, 5)
w = nn.init.eye_(w)

print("w = ", w)
# Dirac delta filler function

'''
6. nn.init.dirac_ - Fills the {3, 4, 5}-dimensional input Tensor with the Dirac delta function.
'''

w = torch.empty(1, 3, 5)
w = nn.init.dirac_(w)

print("w = ", w)
# Xavier uniform filler function

'''
7. nn.init.xavier_uniform_ - Fills the input Tensor with values according to the method described in “Understanding the 
    difficulty of training deep feedforward neural networks” - <NAME>. & <NAME>. (2010), using a uniform distribution.
'''

w = torch.empty(3, 5)
w = nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain('relu'))

print("w = ", w)
# Xavier normal filler function

'''
8. nn.init.xavier_normal_ - Fills the input Tensor with values according to the method described in “Understanding the 
    difficulty of training deep feedforward neural networks” - <NAME>. & <NAME>. (2010), using a normal distribution. 
'''

w = torch.empty(3, 5)
w = nn.init.xavier_normal_(w, gain=nn.init.calculate_gain('leaky_relu'))

print("w = ", w)
# Kaming uniform filler function

'''
9. nn.init.kaiming_uniform_ - Fills the input Tensor with values according to the method described in “Delving deep 
    into rectifiers: Surpassing human-level performance on ImageNet classification” - <NAME> al. (2015), 
    using a uniform distribution. 
            - tensor – an n-dimensional torch.Tensor
            - a – the negative slope of the rectifier used after this layer (0 for ReLU by default)
            - mode – either ‘fan_in’ (default) or ‘fan_out’. Choosing fan_in preserves the magnitude of the variance 
                 of the weights in the forward pass. Choosing fan_out preserves the magnitudes in the backwards pass.
            - nonlinearity – the non-linear function (nn.functional name), recommended to use only with 
                ‘relu’ or ‘leaky_relu’ (default).

'''

w = torch.empty(3, 5)
w = nn.init.kaiming_uniform_(w, mode='fan_in', nonlinearity='relu')

print("w = ", w)
# Kaming normal filler function

'''
10. nn.init.kaiming_normal_ - Fills the input Tensor with values according to the method described in “Delving deep 
        into rectifiers: Surpassing human-level performance on ImageNet classification” - 
        He, K. et al. (2015), using a normal distribution. 
            - tensor – an n-dimensional torch.Tensor
            - a – the negative slope of the rectifier used after this layer (0 for ReLU by default)
            - mode – either ‘fan_in’ (default) or ‘fan_out’. Choosing fan_in preserves the magnitude of the variance 
                 of the weights in the forward pass. Choosing fan_out preserves the magnitudes in the backwards pass.
            - nonlinearity – the non-linear function (nn.functional name), recommended to use only with 
                ‘relu’ or ‘leaky_relu’ (default).

'''

w = torch.empty(3, 5)
w = nn.init.kaiming_normal_(w, mode='fan_in', nonlinearity='relu')

print("w = ", w)
# Orthogonal filler function

'''
11. torch.nn.init.orthogonal_ - Fills the input Tensor with a (semi) orthogonal matrix, as described in 
        “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks” - <NAME>. et al. (2013). 
            - tensor – an n-dimensional torch.Tensor, where n≥2
            - gain – optional scaling factor
'''

w = torch.empty(3, 5)
w = nn.init.orthogonal_(w)

print("w = ", w)

# Sparse filler function

'''
12. nn.init.sparse_ - Fills the 2D input Tensor as a sparse matrix, where the non-zero elements will be drawn 
    from the normal distribution N(0,0.01), as described in “Deep learning via Hessian-free optimization” 
    - <NAME>. (2010).
        - tensor – an n-dimensional torch.Tensor
        - sparsity – The fraction of elements in each column to be set to zero
        - std – the standard deviation of the normal distribution used to generate the non-zero values

'''

w = torch.empty(3, 5)
w = nn.init.sparse_(w, sparsity=0.2)

print("w = ", w)<jupyter_output>w =  tensor([[-0.0088, -0.0039,  0.0000,  0.0053, -0.0024],
        [-0.0053,  0.0000,  0.0062,  0.0000, -0.0075],
        [ 0.0000, -0.0011, -0.0023,  0.0045,  0.0000]])
{'fraction_non_alphanumeric': 0.09222955815134586, 'fraction_numerical': 0.02478415439309294, 'mean_word_length': 2.331979695431472, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '237255', 'license': 'Apache-2.0', 'n_tokens_mistral': 2875, 'n_tokens_neox': 2681, 'n_words': 1020, 'path': 'J) RoadMap 10 - Torch NN 4 - Initializers.ipynb', 'repo_name': 'Charlottecuc/Pytorch_Tutorial', 'size': '9882'}