<commit_before>---
layout: post
title:  "Blog 200: Deep Learning: Natural Language Processing"
date:   2019-07-28 00:11:31 +0530
categories: jekyll update
mathjax: true
---

# Content

1. TOC
{:toc}
---

Quick Refresher: Neural Network in NLP

# Deep Feed Forward Network

- These models are called Feed Forward Network because the information flows from $x$, through the intermediate computations used to define $f$ and finally to the output $y$.
- When FFN are extended to include feedback loops, they are called recurrent neural network.
- The dimensionality of the hidden layer decides the width of the model.
- The strategy of deep learning is to learn $\phi$. In this approach, we have a model $y=f(x;\theta,w)=\phi(x;\theta)^Tx$. We now have $\theta$, that we use to learn $\phi$ from a broad class of functions and parameter $w$ that map from $\phi(x)$ to the desired output. This is an example of deep FNN and $\phi$ denotes the hidden layer.
-  This approach gives up the convexity of the training problem but the benefits out-weight the harm.
- FFN has introduced the concept of **hidden layer**, and this requires us to choose the **activation function** to compute the the hidden layer values.

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Gradient-Based Learning

- The biggest difference between linear models and neural network is that the non-linearity of neural network causes the most interesting loss function to become **non-convex**.
- Neural network models are trained by using iterative, gradient based optimizers, that merely drive the cost function to a very low value.
- Where as linear equation solvers used to train linear regression models or the convex optimization algorithms, with **global convergence guarantees** used to train logistic regression or SVMs.
- SGD applied to non-convex loss function has **no convergence guarantees**, and is sensitive to the initial parameter values.

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Short Summary of RNN, LSTM, GRU

- RNN reads the document left to right and after every word updates the state. By the time it reaches the end of the word, the information obtained from the `first few` words, are completely lost. So we need some mechanism to retain these information.
- Now the most famous rchitecture is `LSTM` to address the above issues of RNN. The general idea is to implement 3 primary operations `selective read`, `selective write` and `selective forget`. i.e forget some old information, create new information and update the current information.
- All the variations of `LSTM` implemented these 3 operations with some tweak, like merging 2 operations into 1 and many more.

## LSTM

- In the most popular version of LSTM, in each LSTM cell these 3 operations are implemented as 3 `Gate` 
  - `selective forget` implemented by `forget gate` $f_t$ 
  - `selective read` implemented by `input gate` $i_t$
  - `selective write` implemented by `output gate` $o_t$

- Each LSTM cell also has 2 states
  - $h_t$: output/hidden state (similar to hidden state of RNN)
  - $C_t$: Cell state or `running state` that acts as a memory containing a fraction of information from all the previous timestamps $t$

### **Gates**

$$f_t = \sigma (W_f  h_{t-1} + U_f x_t + b_f)$$
$$i_t = \sigma (W_i  h_{t-1} + U_i x_t + b_i)$$
$$i_t = \sigma (W_o  h_{t-1} + U_o x_t + b_o)$$

$f_t$, $i_t$, $o_t$ decides the fraction of {$\in [0,1]$} forget, read and write respectively. 

### **States**

$$\tilde{C_t} = \sigma(WH_{t-1} + Ux_t + b)$$
$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C_t}$$
$$h_t = o_t \odot \sigma(C_t)$$

Combining all together:

$$
 \begin{bmatrix} 
  i_t \\ 
  f_t \\
  o_t \\
  \tilde{C_t} 
\end{bmatrix}_{4x1}
 =
  \begin{bmatrix}
   \sigma  \\
   \sigma  \\
   \sigma  \\
   \sigma  
   \end{bmatrix}_{4x1} \odot (W_{4x2}
\begin{bmatrix}
   h_{t-1}  \\
   x_t  
   \end{bmatrix}_{2x1}
   + B_{4x1})
$$

$$
W = \begin{bmatrix} <commit_msg>Typo fixed<commit_after>---
layout: post
title:  "Blog 200: Deep Learning: Natural Language Processing"
date:   2019-07-28 00:11:31 +0530
categories: jekyll update
mathjax: true
---

# Content

1. TOC
{:toc}
---

Quick Refresher: Neural Network in NLP

# Deep Feed Forward Network

- These models are called Feed Forward Network because the information flows from $x$, through the intermediate computations used to define $f$ and finally to the output $y$.
- When FFN are extended to include feedback loops, they are called recurrent neural network.
- The dimensionality of the hidden layer decides the width of the model.
- The strategy of deep learning is to learn $\phi$. In this approach, we have a model $y=f(x;\theta,w)=\phi(x;\theta)^Tx$. We now have $\theta$, that we use to learn $\phi$ from a broad class of functions and parameter $w$ that map from $\phi(x)$ to the desired output. This is an example of deep FNN and $\phi$ denotes the hidden layer.
-  This approach gives up the convexity of the training problem but the benefits out-weight the harm.
- FFN has introduced the concept of **hidden layer**, and this requires us to choose the **activation function** to compute the the hidden layer values.

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Gradient-Based Learning

- The biggest difference between linear models and neural network is that the non-linearity of neural network causes the most interesting loss function to become **non-convex**.
- Neural network models are trained by using iterative, gradient based optimizers, that merely drive the cost function to a very low value.
- Where as linear equation solvers used to train linear regression models or the convex optimization algorithms, with **global convergence guarantees** used to train logistic regression or SVMs.
- SGD applied to non-convex loss function has **no convergence guarantees**, and is sensitive to the initial parameter values.

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Short Summary of RNN, LSTM, GRU

- RNN reads the document left to right and after every word updates the state. By the time it reaches the end of the word, the information obtained from the `first few` words, are completely lost. So we need some mechanism to retain these information.
- Now the most famous architecture is `LSTM` to address the above issues of RNN. The general idea is to implement 3 primary operations `selective read`, `selective write` and `selective forget`. i.e forget some old information, create new information and update the current information.
- All the variations of `LSTM` implemented these 3 operations with some tweak, like merging 2 operations into 1 and many more.

## LSTM

- In the most popular version of LSTM, in each LSTM cell these 3 operations are implemented as 3 `Gate` 
  - `selective forget` implemented by `forget gate` $f_t$ 
  - `selective read` implemented by `input gate` $i_t$
  - `selective write` implemented by `output gate` $o_t$

- Each LSTM cell also has 2 states
  - $h_t$: output/hidden state (similar to hidden state of RNN)
  - $C_t$: Cell state or `running state` that acts as a memory containing a fraction of information from all the previous timestamps $t$

### **Gates**

- Forget Gate:
$$f_t = \sigma (W_f  h_{t-1} + U_f x_t + b_f)$$

- Input / Read Gate
$$i_t = \sigma (W_i  h_{t-1} + U_i x_t + b_i)$$

- Output / Write Gate
  
$$o_t = \sigma (W_o  h_{t-1} + U_o x_t + b_o)$$

$f_t$, $i_t$, $o_t$ decides the fraction of {$\in [0,1]$} forget, read and write respectively. 

### States


- New Content
$$\tilde{C_t} = \sigma(WH_{t-1} + Ux_t + b)$$

- Cell / Running State: Obtained from new content after applying selective forget and read
  
$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C_t}$$

- Output / Hidden state
- 
$$h_t = o_t \odot \sigma(C_t)$$

Combining all together:

$$
 \begin{bmatrix} 
  i_t \\ 
  f_t \\
  o_t \\
  \tilde{C_t} 
\end{bmatrix}_{4x1}
 =
  \begin{bmatrix}
   \sigma  \\
   \sigma  \\
   \sigma  \\
   \sigma  
   \end{bmatrix}_{4x1} \odot (W_{4x2}
\begin{bmatrix}
   h_{t-1}  \\
   x_t  
   \end{bmatrix}_{2x1}
   + B_{4x1})
$$

$$
W = \begin{bmatrix} 
{'fraction_non_alphanumeric': 0.10590656015492617, 'fraction_numerical': 0.014282256112321472, 'mean_word_length': 3.776300578034682, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4874440', 'n_tokens_mistral': 2664, 'n_tokens_neox': 2507, 'n_words': 1141}