<jupyter_start><jupyter_text>Normalization in Neural NetworksIn this notebook I'll discuss the different types of Normalizations that are commonly used in Neural Networks, along with their applications and implementations<jupyter_code>import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers as L<jupyter_output><empty_output><jupyter_text>Batch NormalizationBatch Normalization was first discussed in the paper [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/pdf/1502.03167.pdf)They define Internal Covariate Shift as the change in the distribution of network activations due to the change in network parameters during training. This adversely affects training speed because the later layers have to adapt to the shifted distribution.They proposed that by whitening the inputs to each layer,we would take a step towards achieving the fixed distributions of inputs that would remove the ill effects of the internal covariate shift.Whitening is linearly transforming inputs to have zero mean, unit variance, and be uncorrelated.**The paper introduces Batch Normalization as follows:**1. Normalize each feature independently to have zero mean and unit variance:$$ \hat{x}^{(k)} = \frac{x^{(k)} - E[x^{(k)}]}{\sqrt{Var[x^{(k)}]}} $$where $ x = (x^{(1)}...x^{(d)}) $ is the d-dimensional input.2. The estimates of mean and variance are from the mini-batch for normalization; instead of calculating the mean and variance across the whole dataset.3. Normalizing each feature to zero mean and unit variance could affect what the layer can represent. To overcome this each feature is scaled and shifted by two trained parameters.$$ y^{(k)} = \gamma^{(k)}\hat{x}^{(k)} + \beta^{(k)} $$where $ y^{(k)} $ is the output of the batch normalization layer.4. An exponential moving average of mean and variance is calculated during the training phase and is then used during inference.<jupyter_code>class BatchNormalization(L.Layer):
    def __init__(self, eps=1e-5, momentum=0.1, **kwargs):
        super().__init__(**kwargs)
        self.eps = eps
        self.momentum = momentum
    
    def build(self, input_shape):
        self.exp_mean = self.add_weight(shape=input_shape[-1], initializer='zeros', trainable=False)
        self.exp_var = self.add_weight(shape=input_shape[-1], initializer='ones', trainable=False)
        
        self.scale = self.add_weight(shape=input_shape[-1], initializer='zeros', trainable=True)
        self.shift = self.add_weight(shape=input_shape[-1], initializer='ones', trainable=True)
    
    def call(self, x):
        x_shape = x.shape
        batch_size = x_shape[0]
        channels = x.shape[-1] 
        
        x = tf.reshape(x, (batch_size, -1, channels))
        
        mean = tf.reduce_mean(x, [0, 1])
        mean_x2 = tf.reduce_mean((x ** 2), [0, 1])
        var = mean_x2 - mean ** 2
        
        self.exp_mean = (1 - self.momentum) * self.exp_mean + self.momentum * mean
        self.exp_var = (1 - self.momentum) * self.exp_var + self.momentum * var
        
        mean = self.exp_mean
        var = self.exp_var
        
        x_norm = (x - tf.reshape(mean, (1, 1, -1))) / tf.reshape(tf.sqrt(var + self.eps), (1, 1, -1))
        x_norm = tf.reshape(self.scale, (1, 1, -1)) * x_norm +tf.reshape(self.shift, (1, 1, -1))
        return tf.reshape(x_norm, x_shape)        
x = tf.random.normal((32, 24, 24, 3))
assert BatchNormalization(3)(x).shape == x.shape<jupyter_output>User settings:

   KMP_AFFINITY=granularity=fine,verbose,compact,1,0
   KMP_BLOCKTIME=0
   KMP_SETTINGS=1
   KMP_WARNINGS=0

Effective settings:

   KMP_ABORT_DELAY=0
   KMP_ADAPTIVE_LOCK_PROPS='1,1024'
   KMP_ALIGN_ALLOC=64
   KMP_ALL_THREADPRIVATE=128
   KMP_ATOMIC_MODE=2
   KMP_BLOCKTIME=0
   KMP_CPUINFO_FILE: value is not defined
   KMP_DETERMINISTIC_REDUCTION=false
   KMP_DEVICE_THREAD_LIMIT=2147483647
   KMP_DISP_NUM_BUFFERS=7
   KMP_DUPLICATE_LIB_OK=false
   KMP_ENABLE_TASK_THROTTLING=true
   KMP_FORCE_REDUCTION: value is not defined
   KMP_FOREIGN_THREADS_THREADPRIVATE=true
   KMP_FORKJOIN_BARRIER='2,2'
   KMP_FORKJOIN_BARRIER_PATTERN='hyper,hyper'
   KMP_GTID_MODE=3
   KMP_HANDLE_SIGNALS=false
   KMP_HOT_TEAMS_MAX_LEVEL=1
   KMP_HOT_TEAMS_MODE=0
   KMP_INIT_AT_FORK=true
   KMP_LIBRARY=throughput
   KMP_LOCK_KIND=queuing
   KMP_MALLOC_POOL_INCR=1M
   KMP_NUM_LOCKS_IN_BLOCK=1
   KMP_PLAIN_BARRIER='2,2'
   KMP_PLAIN_BARRIER_PATTERN='hyper,hyper'
   KMP_REDUCTION_BARRIER='1,1'
  [...]<jupyter_text>Layer NormalizationLayer normalization, introduced in the paper [Layer Normalization](https://arxiv.org/pdf/1607.06450.pdf) is a simpler normalization method that is generally used for NLP tasks but works on a wider range of settings.$ LN(x) = \gamma . \frac{X - E_{H, W, C}[X]}{\sqrt{Var_{H, W, C}[x] + \epsilon}} + \beta $<jupyter_code>class LayerNormalization(L.Layer):
    def __init__(self, eps=1e-5,  **kwargs):
        super().__init__(**kwargs)
        self.eps = eps    
        
    def build(self, input_shape):
        self.gain = self.add_weight(shape=input_shape, initializer='zeros', trainable=True)  
        self.bias = self.add_weight(shape=input_shape, initializer='ones', trainable=True)  
    
    def call(self, x):
        normalized_shape = x.shape[1:]
        
        dims = [-(i + 1) for i in range(len(normalized_shape))]
          
        mean = tf.reduce_mean(x, dims, keepdims=True)
        mean_x2 = tf.reduce_mean((x**2), dims, keepdims=True)
        var = mean_x2 - mean ** 2
        
        x_norm = (x - mean) / tf.sqrt(var + self.eps)
        x_norm = self.gain * x_norm + self.bias

        return x_norm
x = tf.random.normal((32, 24, 24, 3))
assert LayerNormalization()(x).shape == x.shape<jupyter_output><empty_output><jupyter_text>Instance NormalizationInstance normalization was introduced in the paper [Instance Normalization: The Missing Ingredient for Fast Stylization](https://arxiv.org/pdf/1607.08022.pdf) to improve style transfer.$ IN(x) = \gamma . \frac{X - E_{H, W}[X]}{\sqrt{Var_{H, W}[x] + \epsilon}} + \beta $<jupyter_code>class InstanceNormalization(L.Layer):
    def __init__(self, eps=1e-5,  **kwargs):
        super().__init__(**kwargs)
        self.eps = eps
        
    def build(self, input_shape):
        self.scale = self.add_weight(shape=input_shape[-1], initializer='zeros', trainable=True)
        self.shift = self.add_weight(shape=input_shape[-1], initializer='ones', trainable=True)
    
    def call(self, x):
        x_shape = x.shape
        batch_size = x_shape[0]
        channels = x.shape[-1] 
        
        x = tf.reshape(x, (batch_size, -1, channels))
        
        mean = tf.reduce_mean(x, [1], keepdims=True)
        mean_x2 = tf.reduce_mean((x ** 2), [1], keepdims=True)
        var = mean_x2 - mean ** 2
        
        x_norm = (x - mean) / tf.sqrt(var + self.eps)
        x_norm = tf.reshape(self.scale, (1, 1, -1)) * x_norm +tf.reshape(self.shift, (1, 1, -1))
        
        return tf.reshape(x_norm, x_shape) 
x = tf.random.normal((32, 24, 24, 3))
assert InstanceNormalization(3)(x).shape == x.shape<jupyter_output><empty_output><jupyter_text>Group NormalizationBatch Normalization works well for large enough batch sizes but not well for small batch sizes, because it normalizes over the batch. Training large models with large batch sizes is not possible due to the memory capacity of the devices.Group Normalization introduced in the paper [Group Normalization](https://arxiv.org/pdf/1803.08494.pdf), normalizes a set of features together as a group. This is based on the observation that classical features such as SIFT and HOG are group-wise features. The paper proposes dividing feature channels into groups and then separately normalizing all channels within each group.All normalization layers can be defined by the following computation.$ \hat{x}_{i} = \frac{1}{\sigma_{i}}(x_{i} - \mu_{i}) $where $ \mu_{i} $ and $ \sigma_{i} $ are mean and standard deviation$ \mu_{i}  = \frac{1}{m}\Sigma_{k \in S_{i}} x_{k} $$ \sigma_{i}  = \sqrt{\frac{1}{m}\Sigma_{k \in S_{i}} (x_{k} - \mu_{i})^{2} + \epsilon} $$ S_{i} $ is the set of indexes across which the mean and standard deviation are calculated for index i. m is the size of the set $ S_{i} $ which is the same for all i.The definition of $ S_{i} $ is different for Batch normalization, Layer normalization, and Instance normalization.**Batch Normalization** $ S_{i} = \{ k|k_{c} = i_{c} \} $ The values that share the same feature channel are normalized together.**Layer Normalization** $ S_{i} = \{ k|k_{n} = i_{n} \} $ The values from the same sample in the batch are normalized together.**Instance Normalization** $ S_{i} = \{ k|k_{n} = i_{n}, k_{c} = i_{c} \} $ The values from the same sample and same feature channel are normalized together.**Group Normalization** $ S_{i} = \{ k|k_{n} = i_{n}, floor(\frac{k_{c}}{C/G}) = floor(\frac{i_{c}}{C/G}) \} $ where $ G $ is the number of groups and $ C $ is the number of channels.Group normalization normalizes values of the same sample and the same group of channels together.<jupyter_code>class GroupNormalization(L.Layer):
    def __init__(self, groups, channels, eps=1e-5,  **kwargs):
        super().__init__(**kwargs)
        self.eps = eps
        self.groups = groups
        self.channels = channels
        
        self.scale = self.add_weight(shape=channels, initializer='zeros', trainable=True)  
        self.shift = self.add_weight(shape=channels, initializer='ones', trainable=True)   
        
    def call(self, x):
        x_shape = x.shape
        batch_size = x_shape[0]
        
        x = tf.reshape(x, (batch_size, -1, self.groups))
        
        mean = tf.reduce_mean(x, [1], keepdims=True)
        mean_x2 = tf.reduce_mean((x ** 2), [1], keepdims=True)
        var = mean_x2 - mean ** 2
        
        x_norm = (x - mean) / tf.sqrt(var + self.eps)
        x_norm = tf.reshape(x_norm, (batch_size, -1, self.channels))
        x_norm = tf.reshape(self.scale, (1, 1, -1)) * x_norm +tf.reshape(self.shift, (1, 1, -1))
        
        return tf.reshape(x_norm, x_shape) 
x = tf.random.normal((32, 24, 24, 8))
assert GroupNormalization(2, 8)(x).shape == x.shape<jupyter_output><empty_output><jupyter_text>Weight StandardizationBatch Normalization doesn't work well when the batch size is too small, which happens when training large networks because of device memory limitations. The paper [Micro-Batch Training with Batch-Channel Normalization and Weight Standardization](https://arxiv.org/pdf/1903.10520.pdf) introduces Weight Standardization with Batch-Channel Normalization as a better alternative. $ \hat{W_{i, j}} = \frac{W_{i, j} - \mu w_{i, .}}{ \sigma w_{i, .} } $ where $ W \in  R^{O \times I} $  $ \mu w_{i, .} = \frac{1}{I} \Sigma_{j=1}^{I} W_{i, j} $  $  \sigma w_{i, .} = \sqrt{\frac{1}{I} \Sigma_{j=1}^{I} W_{i, j}^{2} - \mu w_{i, .}^{2} + \epsilon} $ for a 2D-convolution layer $ O $ is the number of output channels and $ I $ is the number of input channels times the kernel size $ (I = C_{in} \times K_{H} \times K_{W}) $<jupyter_code>def weight_standardization(weight):
    c_out, c_in, *kernel_shape = weight.shape
    weight = tf.reshape(weight, (c_out, -1))
    
    mean = tf.reduce_mean(weight, [1], keepdims=True)
    mean_x2 = tf.reduce_mean((weight ** 2), [1], keepdims=True)
    var = mean_x2 - mean ** 2
    
    weight = (weight - mean) / (tf.sqrt(var + eps))
    
    return tf.reshape(weight, (c_out, c_in, *kernel_shape))<jupyter_output><empty_output><jupyter_text>Batch-Channel NormalizationThis first performs a batch normalizationThis first performs a batch normalization. Then a channel normalization is performed.Channel Normalization is similar to Group Normalization but affine transform is done group wise.<jupyter_code>class ChannelNormalization(L.Layer):
    def __init__(self, groups, channels, eps=1e-5,  **kwargs):
        super().__init__(**kwargs)
        self.eps = eps
        self.groups = groups
        self.channels = channels
        
        self.scale = self.add_weight(shape=groups, initializer='zeros', trainable=True)  
        self.shift = self.add_weight(shape=groups, initializer='ones', trainable=True)          
        
    def call(self, x):
        x_shape = x.shape
        batch_size = x_shape[0]
        
        x = tf.reshape(x, (batch_size, -1, self.groups))
        
        mean = tf.reduce_mean(x, [1], keepdims=True)
        mean_x2 = tf.reduce_mean((x ** 2), [1], keepdims=True)
        var = mean_x2 - mean ** 2
        
        x_norm = (x - mean) / tf.sqrt(var + self.eps)
        x_norm = tf.reshape(self.scale, (1, 1, -1)) * x_norm +tf.reshape(self.shift, (1, 1, -1))
        
        return tf.reshape(x_norm, x_shape) 
x = tf.random.normal((32, 24, 24, 8))
assert ChannelNormalization(2, 8)(x).shape == x.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12630844382414516, 'fraction_numerical': 0.018764053655888967, 'mean_word_length': 3.4738120013874436, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '556212', 'license': 'MIT', 'n_tokens_mistral': 4397, 'n_tokens_neox': 4226, 'n_words': 1173, 'path': 'Normalization-In-Neural-Networks.ipynb', 'repo_name': 'Ritvik19/Implemented-Data-Science', 'size': '12897'}