<commit_before>---
layout: post
title:  "Survey - Residual Network"
date:   2020-03-17 00:00:10 -0030
categories: jekyll update
mathjax: true
---

# Content

1. TOC
{:toc}

---

# Introduction

He et. al in his seminal paper [Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385.pdf) presented this idea of **Residual Network** to overcome the problems of learning with deeper network. The original paper is quite interesting. However for a gist you can follow the great article provided by <NAME> in his course [Dive into Deep Learning, Section 7.6](https://d2l.ai/chapter_convolutional-modern/resnet.html). I am borrowing the content from there for easy understanding. The pytorch code is available in this [github](https://nbviewer.jupyter.org/github/dsgiitr/d2l-pytorch/tree/master/) repo [chapter 9].

However another great tutorial is available from <NAME> in his [github](https://github.com/rasbt/deeplearning-models) repo as well. This survey is a combination of all of them.

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Residual Networks (ResNet)

As we design increasingly deeper networks it becomes imperative to understand how adding layers can increase the `complexity` and `expressiveness` of the network. Even more important is the ability to design networks where adding layers **makes networks strictly more expressive** rather than just different. To make some progress we need a bit of theory.


## Function Classes

Consider $\mathcal{F}$, the class of functions that a specific network architecture (together with learning rates and other hyperparameter settings) can reach. That is, for all $f \in \mathcal{F}$ there exists some set of parameters $W$ that can be obtained through training on a suitable dataset. Let’s assume that $f^∗$ is the function that we really would like to find. 

If it is in $\mathcal{F}$, we are in good shape but typically we will not be quite so lucky. Instead, we will try to find some $f^∗_{\mathcal{F}}$ which is our best bet within $\mathcal{F}$. For instance, we might try finding it by solving the following optimization problem:

$$
f^*_\mathcal{F} := \mathop{\mathrm{argmin}}_f L(X, Y, f) \text{ subject to } f \in \mathcal{F}.
$$


It is only reasonable to assume that if we design a different and more powerful architecture $\mathcal{F}'$ we should arrive at a better outcome. In other words, we would expect that $f_{F'}^{*}$ is **better** than $f_{F}^*$. However, if 

$$\mathcal{F} \nsubseteq \mathcal{F'}$$ 

there is no guarantee that this should even happen. In fact, $f^{∗}_{\mathcal{F}'}$ might well be worse. 

This is a situation that we often encounter in practice - adding layers does not only make the network more expressive, it also changes it in sometimes not quite so predictable ways. The following illustrates this in slightly abstract terms.

<center>
<img src="https://d2l.ai/_images/functionclasses.svg" width="500" alt="image">
</center>
<commit_msg>Update blog on Transformenr<commit_after>---
layout: post
title:  "Survey - Residual Network"
date:   2020-03-17 00:00:10 -0030
categories: jekyll update
mathjax: true
---

# Content

1. TOC
{:toc}

---

# Introduction

He et. al in his seminal paper [Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385.pdf) presented this idea of **Residual Network** to overcome the problems of learning with deeper network. The original paper is quite interesting. However for a gist you can follow the great article provided by <NAME> in his course [Dive into Deep Learning, Section 7.6](https://d2l.ai/chapter_convolutional-modern/resnet.html). I am borrowing the content from there for easy understanding. The pytorch code is available in this [github](https://nbviewer.jupyter.org/github/dsgiitr/d2l-pytorch/tree/master/) repo [chapter 9].

However another great tutorial is available from <NAME> in his [github](https://github.com/rasbt/deeplearning-models) repo as well. This survey is a combination of all of them.

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Residual Networks (ResNet)

As we design increasingly deeper networks it becomes imperative to understand how adding layers can increase the `complexity` and `expressiveness` of the network. Even more important is the ability to design networks where adding layers **makes networks strictly more expressive** rather than just different. To make some progress we need a bit of theory.


## Function Classes

Consider $\mathcal{F}$, the class of functions that a specific network architecture (together with learning rates and other hyperparameter settings) can reach. That is, for all $f \in \mathcal{F}$ there exists some set of parameters $W$ that can be obtained through training on a suitable dataset. Let’s assume that $f^∗$ is the function that we really would like to find. 

If it is in $\mathcal{F}$, we are in good shape but typically we will not be quite so lucky. Instead, we will try to find some $f^∗_{\mathcal{F}}$ which is our best bet within $\mathcal{F}$. For instance, we might try finding it by solving the following optimization problem:

$$
f^*_\mathcal{F} := \mathop{\mathrm{argmin}}_f L(X, Y, f) \text{ subject to } f \in \mathcal{F}.
$$


It is only reasonable to assume that if we design a different and more powerful architecture $\mathcal{F}'$ we should arrive at a better outcome. In other words, we would expect that $f_{F'}^{\star}$ is **better** than $f_{F}^{\star}$. However, if 

$$\mathcal{F} \nsubseteq \mathcal{F'}$$ 

there is no guarantee that this should even happen. In fact, $f^{∗}_{\mathcal{F}'}$ might well be worse. 

This is a situation that we often encounter in practice - adding layers does not only make the network more expressive, it also changes it in sometimes not quite so predictable ways. The following illustrates this in slightly abstract terms.

<center>
<img src="https://d2l.ai/_images/functionclasses.svg" width="500" alt="image">
</center>

{'fraction_non_alphanumeric': 0.10112359550561797, 'fraction_numerical': 0.014210178453403834, 'mean_word_length': 4.522810218978102, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '975000', 'n_tokens_mistral': 1741, 'n_tokens_neox': 1643, 'n_words': 811}