<commit_before>---
layout: post
title:  "Survey - Attention"
date:   2020-03-15 00:00:10 -0030
categories: jekyll update
mathjax: true
---

<link rel="stylesheet" href="/assets/css/markdownstyle.css">

**DISCLAIMER:** I am not the original author of these contents. I compile from various sources to understand a concept and for future reference as I believe in DRY (Don’t Repeat Yourself !!). This blog is nothing but a SCRIBE for me. Solely for education purposes. I try my best to put all the due credits in the reference. If you find any content is incorrect or credit is missing, please contact me. I will be happy to add/edit them. 
{: .olive}

----

# Content

1. TOC
{:toc}

---

# Introduction

From the blog of [<NAME>](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/)

Sequence-to-sequence models are deep learning models that have achieved a lot of success in tasks like machine translation, text summarization, and image captioning. Google Translate started using such a [model](https://blog.google/products/translate/found-translation-more-accurate-fluent-sentences-google-translate/) in production in late 2016. These models are explained in the two pioneering papers ([Sutskever et al. 2014](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf), [Cho et al., 2014](http://emnlp2014.org/papers/pdf/EMNLP2014179.pdf)).

In neural machine translation, a sequence is a series of words, processed one after another. The output is, likewise, a series of words:

<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/seq2seq_2.mp4" frameborder="0" allowfullscreen="true" width="100%" height="150"> </iframe>
</figure>
</center>

Under the hood, the model is composed of an `encoder` and a `decoder`.

The `encoder` processes each item in the input sequence, it compiles the information it captures into a vector (called the `context`). After processing the entire input sequence, the encoder sends the context over to the `decoder`, which begins producing the output sequence item by item.


<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/seq2seq_4.mp4" frameborder="0" allowfullscreen="true" width="100%" height="150"> </iframe>
</figure>
</center>

The context is nothing but a **vector of floats**.  It is basically the number of hidden units in the encoder RNN. 


The `context` vector turned out to be a bottleneck for these types of models. It made it challenging for the models to deal with long sentences. A solution was proposed in [Bahdanau et al., 2014](https://arxiv.org/abs/1409.0473) and [Luong et al., 2015](https://arxiv.org/abs/1508.04025). These papers introduced and refined a technique called **Attention**, which highly improved the quality of machine translation systems. Attention allows the model to focus on the relevant parts of the input sequence as needed.

An attention model differs from a classic sequence-to-sequence model in two main ways:

- First, the `encoder` passes a lot more data to the `decoder`. Instead of passing the last hidden state of the encoding stage, the encoder **passes all the hidden states to the decoder**:


<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/seq2seq_7.mp4" frameborder="0" allowfullscreen="true" width="100%" height="300"> </iframe>
</figure>
</center>


**Attention mechanism takes a linear combination of all the encoder hidden states**
{: .red}


- Second, an attention decoder does an extra step before producing its output. In order to focus on the parts of the input that are relevant to this decoding time step, the decoder does the following:

- Look at the set of encoder hidden states it received – each encoder hidden states is most associated with a certain word in the input sentence
- Give each hidden states a score (let’s ignore how the scoring is done for now)
- Multiply each hidden states by its softmaxed score, thus amplifying hidden states with high scores, and drowning out hidden states with low scores

<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/attention_process.mp4" frameborder="0" allowfullscreen="true" width="100%" height="350"> </iframe>
</figure>
</center>


<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/attention_tensor_dance.mp4" frameborder="0" allowfullscreen="true" width="100%" height="350"> </iframe>
</figure>
</center>


>  The attention-mechanism looks at an input sequence and decides at each step which other parts of the sequence are important.


Neural networks, in particular recurrent neural networks (RNNs), are now at the core of the leading approaches to language understanding tasks such as **language modeling, machine translation and question answering**. In [Attention Is All You Need](https://arxiv.org/abs/1706.03762), the authors introduce the **Transformer**, a novel neural network architecture based on a self-attention mechanism that we believe to be particularly well suited for language understanding.

In the paper, the author shows that the Transformer outperforms both recurrent and convolutional models on academic `English to German` and `English to French` translation benchmarks. On top of higher translation quality, the Transformer requires **less computation to train** and is a much better fit for modern machine learning hardware, speeding up training by up to an order of magnitude.

The **sequence-to-sequence** neural network models are widely used for NLP. A popular type of these models is an **encoder-decoder**. There, one part of the network — encoder — encodes the input sequence into a fixed-length context vector. This vector is an internal representation of the text. This context vector is then decoded into the output sequence by the decoder. See an example:

<center>
<img src="https://miro.medium.com/max/900/1*1ui7iDq956eDs-mAZHEdIg.png" width="600">
</center>


However, there is a catch with the common encoder-decoder approach: a neural network compresses all the information of an input source sentence into a fixed-length vector. It has been shown that this leads to a decline in performance when dealing with long sentences. The attention mechanism was introduced by Bahdanau in “Neural Machine Translation by Jointly Learning to Align and Translate” to alleviate this problem.

:paperclip: **Reference:**

- [Attention Explained by <NAME>](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/) :fire: :fire: MUST read

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Transformer - Visual Understanding

Must Watch :fire:

<center>

<figure class="video_container">
  <iframe src="https://www.youtube.com/watch?v=mV8c9dJpxQE" frameborder="0" allowfullscreen="true" width="100%" height="300"> </iframe>
</figure>

</center>

_*If above link is broken, click [here](https://www.youtube.com/watch?v=mV8c9dJpxQE)_

<iframe  title="YouTube video player" width="480" height="390" src="https://www.youtube.com/watch?v=mV8c9dJpxQE" frameborder="0" allowfullscreen></iframe>


- The Transformer uses attention to boost the speed with which these models can be trained.
- The biggest benefit, however, comes from how The Transformer lends itself to **parallelization**.
- At the core of Transformer, there is `encoder-decoder` block

The encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements). The decoding component is a stack of decoders of the same number.

<center>
<img src="https://jalammar.github.io/images/t/The_transformer_encoder_decoder_stack.png" width="500" alt="image">
</center>

**The encoders are all identical in structure (yet they do not share weights)**.
{: .purple}

- Each `encoder` block is a combination of a FFNN and Self-Attention


<center>
<img src="https://jalammar.github.io/images/t/Transformer_encoder.png" width="500" alt="image">
</center>
<commit_msg>Update latex issue<commit_after>---
layout: post
title:  "Survey - Attention"
date:   2020-03-15 00:00:10 -0030
categories: jekyll update
mathjax: true
---

<link rel="stylesheet" href="/assets/css/markdownstyle.css">

**DISCLAIMER:** I am not the original author of these contents. I compile from various sources to understand a concept and for future reference as I believe in DRY (Don’t Repeat Yourself !!). This blog is nothing but a SCRIBE for me. Solely for education purposes. I try my best to put all the due credits in the reference. If you find any content is incorrect or credit is missing, please contact me. I will be happy to add/edit them. 
{: .olive}

----

# Content

1. TOC
{:toc}

---

# Introduction

From the blog of [<NAME>](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/)

Sequence-to-sequence models are deep learning models that have achieved a lot of success in tasks like machine translation, text summarization, and image captioning. Google Translate started using such a [model](https://blog.google/products/translate/found-translation-more-accurate-fluent-sentences-google-translate/) in production in late 2016. These models are explained in the two pioneering papers ([Sutskever et al. 2014](https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf), [Cho et al., 2014](http://emnlp2014.org/papers/pdf/EMNLP2014179.pdf)).

In neural machine translation, a sequence is a series of words, processed one after another. The output is, likewise, a series of words:

<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/seq2seq_2.mp4" frameborder="0" allowfullscreen="true" width="100%" height="150"> </iframe>
</figure>
</center>

Under the hood, the model is composed of an `encoder` and a `decoder`.

The `encoder` processes each item in the input sequence, it compiles the information it captures into a vector (called the `context`). After processing the entire input sequence, the encoder sends the context over to the `decoder`, which begins producing the output sequence item by item.


<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/seq2seq_4.mp4" frameborder="0" allowfullscreen="true" width="100%" height="150"> </iframe>
</figure>
</center>

The context is nothing but a **vector of floats**.  It is basically the number of hidden units in the encoder RNN. 


The `context` vector turned out to be a bottleneck for these types of models. It made it challenging for the models to deal with long sentences. A solution was proposed in [Bahdanau et al., 2014](https://arxiv.org/abs/1409.0473) and [Luong et al., 2015](https://arxiv.org/abs/1508.04025). These papers introduced and refined a technique called **Attention**, which highly improved the quality of machine translation systems. Attention allows the model to focus on the relevant parts of the input sequence as needed.

An attention model differs from a classic sequence-to-sequence model in two main ways:

- First, the `encoder` passes a lot more data to the `decoder`. Instead of passing the last hidden state of the encoding stage, the encoder **passes all the hidden states to the decoder**:


<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/seq2seq_7.mp4" frameborder="0" allowfullscreen="true" width="100%" height="300"> </iframe>
</figure>
</center>


**Attention mechanism takes a linear combination of all the encoder hidden states**
{: .red}


- Second, an attention decoder does an extra step before producing its output. In order to focus on the parts of the input that are relevant to this decoding time step, the decoder does the following:

- Look at the set of encoder hidden states it received – each encoder hidden states is most associated with a certain word in the input sentence
- Give each hidden states a score (let’s ignore how the scoring is done for now)
- Multiply each hidden states by its softmaxed score, thus amplifying hidden states with high scores, and drowning out hidden states with low scores

<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/attention_process.mp4" frameborder="0" allowfullscreen="true" width="100%" height="350"> </iframe>
</figure>
</center>


<center>
<figure class="video_container">
  <iframe src="https://jalammar.github.io/images/attention_tensor_dance.mp4" frameborder="0" allowfullscreen="true" width="100%" height="350"> </iframe>
</figure>
</center>


>  The attention-mechanism looks at an input sequence and decides at each step which other parts of the sequence are important.


Neural networks, in particular recurrent neural networks (RNNs), are now at the core of the leading approaches to language understanding tasks such as **language modeling, machine translation and question answering**. In [Attention Is All You Need](https://arxiv.org/abs/1706.03762), the authors introduce the **Transformer**, a novel neural network architecture based on a self-attention mechanism that we believe to be particularly well suited for language understanding.

In the paper, the author shows that the Transformer outperforms both recurrent and convolutional models on academic `English to German` and `English to French` translation benchmarks. On top of higher translation quality, the Transformer requires **less computation to train** and is a much better fit for modern machine learning hardware, speeding up training by up to an order of magnitude.

The **sequence-to-sequence** neural network models are widely used for NLP. A popular type of these models is an **encoder-decoder**. There, one part of the network — encoder — encodes the input sequence into a fixed-length context vector. This vector is an internal representation of the text. This context vector is then decoded into the output sequence by the decoder. See an example:

<center>
<img src="https://miro.medium.com/max/900/1*1ui7iDq956eDs-mAZHEdIg.png" width="600">
</center>


However, there is a catch with the common encoder-decoder approach: a neural network compresses all the information of an input source sentence into a fixed-length vector. It has been shown that this leads to a decline in performance when dealing with long sentences. The attention mechanism was introduced by Bahdanau in “Neural Machine Translation by Jointly Learning to Align and Translate” to alleviate this problem.

:paperclip: **Reference:**

- [Attention Explained by <NAME>](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/) :fire: :fire: MUST read

<a href="#Top" style="color:#2F4F4F;background-color: #c8f7e4;float: right;">Content</a>

----

# Transformer - Visual Understanding

Must Watch :fire:

<center>

<figure class="video_container">
  <iframe src="https://www.youtube.com/watch?v=mV8c9dJpxQE" frameborder="0" allowfullscreen="true" width="100%" height="300"> </iframe>
</figure>

</center>

_*If above link is broken, click [here](https://www.youtube.com/watch?v=mV8c9dJpxQE)_

<iframe  title="YouTube video player" width="480" height="390" src="https://www.youtube.com/watch?v=mV8c9dJpxQE" frameborder="0" allowfullscreen></iframe>

<object data="https://www.youtube.com/watch?v=mV8c9dJpxQE"
   width="560" height="315"></object>

- The Transformer uses attention to boost the speed with which these models can be trained.
- The biggest benefit, however, comes from how The Transformer lends itself to **parallelization**.
- At the core of Transformer, there is `encoder-decoder` block

The encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements). The decoding component is a stack of decoders of the same number.

<center>
<img src="https://jalammar.github.io/images/t/The_transformer_encoder_decoder_stack.png" width="500" alt="image">
</center>

**The encoders are all identical in structure (yet they do not share weights)**.
{: .purple}

- Each `encoder` block is a combination of a FFNN and Self-Attention


<center>
<img src="https://jalammar.github.io/images/t/Transformer_encoder.png" width="500" alt="image">
</center>

{'fraction_non_alphanumeric': 0.09863411096337484, 'fraction_numerical': 0.021274023933276924, 'mean_word_length': 4.965032444124009, 'pattern_counts': {'":': 0, '<': 109, '<?xml version=': 0, '>': 111, 'https://': 37, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5099121', 'n_tokens_mistral': 4872, 'n_tokens_neox': 4458, 'n_words': 1996}