<commit_before>---
title: "Why Batch Normalization Works so Well?"
collection: projects
type: "Final project"
permalink: /projects/1_2017-07-01-BN_cmc
date: 2017-07-01
venue: 'Machine Learning having it deep and structured, NTU, Spring 2017'
paperurl: 'http://nothinghard.github.io/files/BN_cmc.pdf'
display: '../images/bn_singular.png'
---

**Batch normalization** (BN) is regarded as the necessary component in many well-known neural network architectures to speed up network training phrase and also to improve performance. In this work, we investigate the effectiveness of BN by comprehensive experiments and also reveal the underlying reasons, in the theoretical point of view, that **why batch normalization works so well**. We have demonstrate the following effects:
- BN does **regularize** the growth of weights (verifed by checking weight distribution)
- BN does **tackle gradient vanishing** problem (verified by monitoring gradient's magnitude)
- BN does **improve gradient flows** of network (verified by examining the isometry of backward propagation)
- We compare the distribution of **singular values of layer's Jacobian** to verify the claim that BN leads to better isometry when error propagates through networks.
- Github repository >> [https://github.com/NothingHard/MLDS_final_2017](https://github.com/NothingHard/MLDS_final_2017)

## Slideshare
<iframe src="//www.slideshare.net/slideshow/embed_code/key/GDTNdQPxwOQ02I" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/ssuser950871/why-batch-normalization-works-so-well" title="Why Batch Normalization Works so Well" target="_blank">Why Batch Normalization Works so Well</a> </strong> from <strong><a href="https://www.slideshare.net/ssuser950871" target="_blank"><NAME></a></strong> </div>

## Why Needs Batch Normalization?
Batch normalization is to accelerate network training by reducing internal covariate shift. **Internal covariate shift** is the change in the distribution of network activations due to the change in network parameters during training. By whitening the inputs to each layer, we would take a step towards achieving the fixed distributions of inputs that would remove the ill effects of the internal covariate shift.

The full whitening of each layer’s inputs is costly and not everywhere differentiable, so, in batch normalization, there are two necessary simplifications:
1. instead of whitening the features in layer inputs and outputs jointly, **independently normalize** each scalar feature
2. use **mini-batches statistics** rather than global statistics in stochastic gradient training

## Experimental Study

We conduct experiments on MNIST using a simple fully connected neural network (3 layers), and 

## Default setting of our model

| Parameter | Selection |
|-----------|-----------|
|  |  |

## Validation of the claims made by the authors
### 1. BN regularizes the model
...
### 2. BN solves the gradient vanishing problem
...
<img src="../images/bn_exp456.png" style="width:100%;">

### 3. BN benefits gradient flows through network
- The distribution of singular values of layer's Jacobian

<img src="../images/l1_relu.gif" style="width:48%;" loop="-1">
<img src="../images/l2_sigmoid.gif" style="width:48%;" loop="-1">

## Which the following factors affect the effect of BN most?

| Experiment | Alternatives |
|------------|--------------|
| 1. Activation function |  lrelu, relu sigmoid, softplus, tanh |
| 2. Optimizer  | Adam, SGD |
| 3. Batch size | 4, 16, 64, 256 |

<img src="../images/bn_exp123.png" style="width:100%;">


## 

## Summary II
<commit_msg>re-organize BN<commit_after>---
title: "Why Batch Normalization Works so Well?"
collection: projects
type: "Final project"
permalink: /projects/1_2017-07-01-BN_cmc
date: 2017-07-01
venue: 'Machine Learning having it Deep and Structured (MLDS), NTU, Spring 2017'
paperurl: 'http://nothinghard.github.io/files/BN_cmc.pdf'
display: '../images/bn_singular.png'
---

Since **batch normalization** (BN) is regarded as the necessary component in many well-known network architectures, this work aims at investigating that **why batch normalization works so well** in both theoretical and experimental viewpoints.
- The 1st place by peer reviews (original score: 16.2/17)
	- originality(5), reproducibility(5), quality(5), novelty(2, bonus)
- The 2nd place in the oral presentation by audience votes
- Github repository >> [https://github.com/NothingHard/MLDS_final_2017](https://github.com/NothingHard/MLDS_final_2017)

## Abstract
BN is designed to reduce internal covariate shift problem in training networks. In this project, we have validated the effectiveness of BN to accelerate network training and improve performance. Furthermore, the experimental results demonstrate the following effects:
- BN does **regularize** the growth of weights (verifed by checking weight distribution)
- BN does **tackle gradient vanishing** problem (verified by monitoring gradient's magnitude)
- BN does **improve gradient flows** of network (verified by examining the isometry of backward propagation)
- We compare the distribution of **singular values of layer's Jacobian** to verify the claim that BN leads to better isometry when error propagates through networks.

## Why Needs Batch Normalization?
Batch normalization is to accelerate network training by reducing internal covariate shift. **Internal covariate shift** is the change in the distribution of network activations due to the change in network parameters during training. By whitening the inputs to each layer, we would take a step towards achieving the fixed distributions of inputs that would remove the ill effects of the internal covariate shift.

The full whitening of each layer’s inputs is costly and not everywhere differentiable, so, in batch normalization, there are two necessary simplifications:
1. instead of whitening the features in layer inputs and outputs jointly, **independently normalize** each scalar feature
2. use **mini-batches statistics** rather than global statistics in stochastic gradient training

## Experimental Study

We conduct experiments on MNIST using a simple fully connected neural network (3 layers). 

| Parameter | Selection |
|-----------|-----------|
|  |  |

## Validation of the claims made by the authors
### 1. BN regularizes the model
...
### 2. BN solves the gradient vanishing problem
...
<img src="../images/bn_exp456.png" style="width:100%;">

### 3. BN benefits gradient flows through network
- The distribution of singular values of layer's Jacobian

<img src="../images/l1_relu.gif" style="width:48%;" loop="-1">
<img src="../images/l2_sigmoid.gif" style="width:48%;" loop="-1">

## Which the following factors affect the effect of BN most?

| Experiment | Alternatives |
|------------|--------------|
| 1. Activation function |  lrelu, relu sigmoid, softplus, tanh |
| 2. Optimizer  | Adam, SGD |
| 3. Batch size | 4, 16, 64, 256 |

<img src="../images/bn_exp123.png" style="width:100%;">


## 

## Slideshare
<iframe src="//www.slideshare.net/slideshow/embed_code/key/GDTNdQPxwOQ02I" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/ssuser950871/why-batch-normalization-works-so-well" title="Why Batch Normalization Works so Well" target="_blank">Why Batch Normalization Works so Well</a> </strong> from <strong><a href="https://www.slideshare.net/ssuser950871" target="_blank"><NAME></a></strong> </div>

{'fraction_non_alphanumeric': 0.11742182514358647, 'fraction_numerical': 0.025909380982769622, 'mean_word_length': 4.800148038490008, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 41, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1762092', 'n_tokens_mistral': 2325, 'n_tokens_neox': 2074, 'n_words': 878}