<commit_before>---
layout: post
comments: true
title: "Generalized Language Models"
date: 2019-01-31 12:00:00
tags: nlp
image: "elmo-and-bert.png"
---

> As a follow up of word embedding post, we will discuss the models on learning contextualized word vectors, as well as the new trend in large unsupervised pre-trained language models which have achieved amazing SOTA results on a variety of language tasks.

<!--more-->

<br />
![Elmo & Bert]({{ '/assets/images/elmo-and-bert.png' | relative_url }})
{: style="width: 60%;" class="center"}
*Fig. 0. I guess they are Elmo & Bert? (Image source: [here](https://www.youtube.com/watch?v=l5einDQ-Ttc))*
<br />

We have seen amazing progress in NLP in 2018. Large-scale pre-trained language modes like [OpenAI GPT](https://blog.openai.com/language-unsupervised/) and [BERT](https://arxiv.org/abs/1810.04805) have achieved great performance on a variety of language tasks using generic model architectures. The idea is similar to how ImageNet classification pre-training helps many vision tasks (\*). Even better than vision classification pre-training, this simple and powerful approach in NLP does not require labeled data for pre-training, allowing us to experiment with increased training scale, up to our very limit.

*(\*) Although recently He et al. (2018) [found](https://arxiv.org/abs/1811.08883) that pre-training might not be necessary for image segmentation task.*

In my previous NLP [post on word embedding]({{ site.baseurl }}{% post_url 2017-10-15-learning-word-embedding %}), the introduced embeddings are not context-specific --- they are learned based on word concurrency but not sequential context. So in two sentences, "*I am eating an apple*" and "*I have an Apple phone*", two "apple" words refer to very different things but they would still share the same word embedding vector. 

Despite this, early adoption of word embeddings in problem-solving is to use them as additional features for an existing task-specific model and in a way the improvement is bounded.

In this post, we will discuss how various approaches were proposed to make embeddings dependent on context, and to make them easier and cheaper to be applied to downstream tasks in general form.


{: class="table-of-content"}
* TOC
{:toc}


## CoVe

**CoVe** ([McCann et al. 2017](https://arxiv.org/abs/1708.00107)), short for **Contextual Word Vectors**, is a type of word embeddings learned by an encoder in an [attentional seq-to-seq]({{ site.baseurl }}{% post_url 2018-06-24-attention-attention %}#born-for-translation) machine translation model.
Different from traditional word embeddings introduced [here]({{ site.baseurl }}{% post_url 2017-10-15-learning-word-embedding %}), CoVe word representations are functions of the entire input sentence.


### NMT Recap

Here the Neural Machine Translation ([NMT](https://github.com/THUNLP-MT/MT-Reading-List)) model is composed of a standard, two-layer, bidirectional LSTM encoder and an attentional two-layer unidirectional LSTM decoder. It is pre-trained on the English-German translation task. The encoder learns and optimizes the embedding vectors of English words in order to translate them to German. With the intuition that the encoder should capture high-level semantic and syntactic meanings before transforming words into another language, the encoder output is used to provide contextualized word embeddings for various downstream language tasks.


![NMT Recap]({{ '/assets/images/nmt-recap.png' | relative_url }})
{: style="width: 100%;" class="center"}
*Fig. 1. The NMT base model used in CoVe.*

- A sequence of $$n$$ words in source language (English): $$x = [x_1, \dots, x_n]$$.
- A sequence of $$m$$ words in target language (German): $$y = [y_1, \dots, y_m]$$.
- The [GloVe]({{ site.baseurl }}{% post_url 2017-10-15-learning-word-embedding %}#glove-global-vectors) vectors of source words: $$\text{GloVe}(x)$$.
- Randomly initialized embedding vectors of target words: $$z = [z_1, \dots, z_m]$$.
- The biLSTM encoder outputs a sequence of hidden states: $$h = [h_1, \dots, h_n] = \text{biLSTM}(\text{GloVe}(x))$$ and $$h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t]$$ where the forward LSTM computes $$\overrightarrow{h}_t = \text{LSTM}(x_t, \overrightarrow{h}_{t-1})$$ and the backward computation gives us $$\overleftarrow{h}_t = \text{LSTM}(x_t, \overleftarrow{h}_{t-1})$$.
- The attentional decoder outputs a distribution over words: $$p(y_t \mid H, y_1, \dots, y_{t-1})$$ where $$H$$ is a stack of hidden states $$\{h\}$$ along the time dimension:


$$
\begin{aligned}
\text{decoder hidden state: } s_t &= \text{LSTM}([z_{t-1}; \tilde{h}_{t-1}], s_{t-1}) \\
\text{attention weights: } \alpha_t &= \text{softmax}(H(W_1 s_t + b_1)) \\
\text{context-adjusted hidden state: } \tilde{h}_t &= \tanh(W_2[H^\top\alpha_t;s_t] + b_2) \\
\text{decoder output: } p(y_t\mid H, y_1, \dots, y_{t-1}) &= \text{softmax}(W_\text{out} \tilde{h}_t + b_\text{out})
\end{aligned}
$$


### Use CoVe in Downstream Tasks

The hidden states of NMT encoder are defined as **context vectors** for other language tasks: 

$$
\text{CoVe}(x) = \text{biLSTM}(\text{GloVe}(x))
$$ 

The paper proposed to use the concatenation of GloVe and CoVe for question-answering and classification tasks. GloVe learns from the ratios of global word co-occurrences, so it has no sentence context, while CoVe is generated by processing text sequences is able to capture the contextual information.

$$
v = [\text{GloVe}(x); \text{CoVe}(x)]
$$

Given a downstream task, we first generate the concatenation of GloVe + CoVe vectors of input words and then feed them into the task-specific models as additional features.

![CoVe model]({{ '/assets/images/CoVe.png' | relative_url }})
{: style="width: 100%;" class="center"}
*Fig. 2. The CoVe embeddings are generated by an encoder trained for machine translation task. The encoder can be plugged into any downstream task-specific model. (Image source: [original paper](https://arxiv.org/abs/1708.00107))*


**Summary**: The limitation of CoVe is obvious: (1) pre-training is bounded by available datasets on the supervised translation task; (2) the contribution of CoVe to the final performance is constrained by the task-specific model architecture.

In the following sections, we will see that ELMo overcomes issue (1) by unsupervised pre-training and OpenAI GPT & BERT further overcome both problems by unsupervised pre-training + using generative model architecture for different downstream tasks.



## ELMo

**ELMo**, short for **Embeddings from Language Model** ([Peters, et al, 2018](https://arxiv.org/abs/1802.05365)) learns contextualized word representation by pre-training a language model in an *unsupervised* way.


### Bidirectional Language Model

The bidirectional Language Model (**biLM**) is the foundation for ELMo. While the input is a sequence of $$n$$ tokens, $$(x_1, \dots, x_n)$$, the language model learns to predict the probability of next token given the history.

In the forward pass, the history contains words before the target token,

$$
p(x_1, \dots, x_n) = \prod_{i=1}^n p(x_i \mid x_1, \dots, x_{i-1})
$$

In the backward pass, the history contains words after the target token,

$$
p(x_1, \dots, x_n) = \prod_{i=1}^n p(x_i \mid x_{i+1}, \dots, x_n)
$$

The predictions in both directions are modeled by multi-layer LSTMs with hidden states $$\overrightarrow{\mathbf{h}}_{i,\ell}$$ and $$\overleftarrow{\mathbf{h}}_{i,\ell}$$ for input token $$x_i$$ at the layer level $$\ell=1,\dots,L$$.
The final layer’s hidden state $$\mathbf{h}_{i,L} = [\overrightarrow{\mathbf{h}}_{i,L}; \overleftarrow{\mathbf{h}}_{i,L}]$$ is used to output the probabilities over tokens after softmax normalization. They share the embedding layer and the softmax layer, parameterized by $$\Theta_e$$ and $$\Theta_s$$ respectively.



![ELMo biLSTM]({{ '/assets/images/ELMo-biLSTM.png' | relative_url }})
{: style="width: 85%;" class="center"}
*Fig. 3. The biLSTM base model of ELMo. (Image source: recreated based on [this](https://towardsdatascience.com/introduction-to-sequence-models-rnn-bidirectional-rnn-lstm-gru-73927ec9df15).)*

The model is trained to minimize the negative log likelihood (= maximize the log likelihood for true words) in both directions:

$$
\begin{aligned}
\mathcal{L} = - \sum_{i=1}^n \Big( 
\log p(x_i \mid x_1, \dots, x_{i-1}; \Theta_e, \overrightarrow{\Theta}_\text{LSTM}, \Theta_s) + \\
\log p(x_i \mid x_{i+1}, \dots, x_n; \Theta_e, \overleftarrow{\Theta}_\text{LSTM}, \Theta_s) \Big)
\end{aligned}
$$


### ELMo Representations

On top of a $$L$$-layer biLM, ELMo stacks all the hidden states across layers together by learning a task-specific linear combination. The hidden state representation for the token $$x_i$$ contains $$2L+1$$ vectors:

$$
R_i = \{ \mathbf{h}_{i,\ell} \mid \ell = 0, \dots, L \}
$$
where $$\mathbf{h}_{0, \ell}$$ is the embedding layer output and $$\mathbf{h}_{i, \ell} = [\overrightarrow{\mathbf{h}}_{i,\ell}; \overleftarrow{\mathbf{h}}_{i,\ell}]$$.<commit_msg>use the correct figure citation<commit_after>---
layout: post
comments: true
title: "Generalized Language Models"
date: 2019-01-31 12:00:00
tags: nlp
image: "elmo-and-bert.png"
---

> As a follow up of word embedding post, we will discuss the models on learning contextualized word vectors, as well as the new trend in large unsupervised pre-trained language models which have achieved amazing SOTA results on a variety of language tasks.

<!--more-->

<br />
![Elmo & Bert]({{ '/assets/images/elmo-and-bert.png' | relative_url }})
{: style="width: 60%;" class="center"}
*Fig. 0. I guess they are Elmo & Bert? (Image source: [here](https://www.youtube.com/watch?v=l5einDQ-Ttc))*
<br />

We have seen amazing progress in NLP in 2018. Large-scale pre-trained language modes like [OpenAI GPT](https://blog.openai.com/language-unsupervised/) and [BERT](https://arxiv.org/abs/1810.04805) have achieved great performance on a variety of language tasks using generic model architectures. The idea is similar to how ImageNet classification pre-training helps many vision tasks (\*). Even better than vision classification pre-training, this simple and powerful approach in NLP does not require labeled data for pre-training, allowing us to experiment with increased training scale, up to our very limit.

*(\*) Although recently He et al. (2018) [found](https://arxiv.org/abs/1811.08883) that pre-training might not be necessary for image segmentation task.*

In my previous NLP [post on word embedding]({{ site.baseurl }}{% post_url 2017-10-15-learning-word-embedding %}), the introduced embeddings are not context-specific --- they are learned based on word concurrency but not sequential context. So in two sentences, "*I am eating an apple*" and "*I have an Apple phone*", two "apple" words refer to very different things but they would still share the same word embedding vector. 

Despite this, early adoption of word embeddings in problem-solving is to use them as additional features for an existing task-specific model and in a way the improvement is bounded.

In this post, we will discuss how various approaches were proposed to make embeddings dependent on context, and to make them easier and cheaper to be applied to downstream tasks in general form.


{: class="table-of-content"}
* TOC
{:toc}


## CoVe

**CoVe** ([McCann et al. 2017](https://arxiv.org/abs/1708.00107)), short for **Contextual Word Vectors**, is a type of word embeddings learned by an encoder in an [attentional seq-to-seq]({{ site.baseurl }}{% post_url 2018-06-24-attention-attention %}#born-for-translation) machine translation model.
Different from traditional word embeddings introduced [here]({{ site.baseurl }}{% post_url 2017-10-15-learning-word-embedding %}), CoVe word representations are functions of the entire input sentence.


### NMT Recap

Here the Neural Machine Translation ([NMT](https://github.com/THUNLP-MT/MT-Reading-List)) model is composed of a standard, two-layer, bidirectional LSTM encoder and an attentional two-layer unidirectional LSTM decoder. It is pre-trained on the English-German translation task. The encoder learns and optimizes the embedding vectors of English words in order to translate them to German. With the intuition that the encoder should capture high-level semantic and syntactic meanings before transforming words into another language, the encoder output is used to provide contextualized word embeddings for various downstream language tasks.


![NMT Recap]({{ '/assets/images/nmt-recap.png' | relative_url }})
{: style="width: 100%;" class="center"}
*Fig. 1. The NMT base model used in CoVe.*

- A sequence of $$n$$ words in source language (English): $$x = [x_1, \dots, x_n]$$.
- A sequence of $$m$$ words in target language (German): $$y = [y_1, \dots, y_m]$$.
- The [GloVe]({{ site.baseurl }}{% post_url 2017-10-15-learning-word-embedding %}#glove-global-vectors) vectors of source words: $$\text{GloVe}(x)$$.
- Randomly initialized embedding vectors of target words: $$z = [z_1, \dots, z_m]$$.
- The biLSTM encoder outputs a sequence of hidden states: $$h = [h_1, \dots, h_n] = \text{biLSTM}(\text{GloVe}(x))$$ and $$h_t = [\overrightarrow{h}_t; \overleftarrow{h}_t]$$ where the forward LSTM computes $$\overrightarrow{h}_t = \text{LSTM}(x_t, \overrightarrow{h}_{t-1})$$ and the backward computation gives us $$\overleftarrow{h}_t = \text{LSTM}(x_t, \overleftarrow{h}_{t-1})$$.
- The attentional decoder outputs a distribution over words: $$p(y_t \mid H, y_1, \dots, y_{t-1})$$ where $$H$$ is a stack of hidden states $$\{h\}$$ along the time dimension:


$$
\begin{aligned}
\text{decoder hidden state: } s_t &= \text{LSTM}([z_{t-1}; \tilde{h}_{t-1}], s_{t-1}) \\
\text{attention weights: } \alpha_t &= \text{softmax}(H(W_1 s_t + b_1)) \\
\text{context-adjusted hidden state: } \tilde{h}_t &= \tanh(W_2[H^\top\alpha_t;s_t] + b_2) \\
\text{decoder output: } p(y_t\mid H, y_1, \dots, y_{t-1}) &= \text{softmax}(W_\text{out} \tilde{h}_t + b_\text{out})
\end{aligned}
$$


### Use CoVe in Downstream Tasks

The hidden states of NMT encoder are defined as **context vectors** for other language tasks: 

$$
\text{CoVe}(x) = \text{biLSTM}(\text{GloVe}(x))
$$ 

The paper proposed to use the concatenation of GloVe and CoVe for question-answering and classification tasks. GloVe learns from the ratios of global word co-occurrences, so it has no sentence context, while CoVe is generated by processing text sequences is able to capture the contextual information.

$$
v = [\text{GloVe}(x); \text{CoVe}(x)]
$$

Given a downstream task, we first generate the concatenation of GloVe + CoVe vectors of input words and then feed them into the task-specific models as additional features.

![CoVe model]({{ '/assets/images/CoVe.png' | relative_url }})
{: style="width: 100%;" class="center"}
*Fig. 2. The CoVe embeddings are generated by an encoder trained for machine translation task. The encoder can be plugged into any downstream task-specific model. (Image source: [original paper](https://arxiv.org/abs/1708.00107))*


**Summary**: The limitation of CoVe is obvious: (1) pre-training is bounded by available datasets on the supervised translation task; (2) the contribution of CoVe to the final performance is constrained by the task-specific model architecture.

In the following sections, we will see that ELMo overcomes issue (1) by unsupervised pre-training and OpenAI GPT & BERT further overcome both problems by unsupervised pre-training + using generative model architecture for different downstream tasks.



## ELMo

**ELMo**, short for **Embeddings from Language Model** ([Peters, et al, 2018](https://arxiv.org/abs/1802.05365)) learns contextualized word representation by pre-training a language model in an *unsupervised* way.


### Bidirectional Language Model

The bidirectional Language Model (**biLM**) is the foundation for ELMo. While the input is a sequence of $$n$$ tokens, $$(x_1, \dots, x_n)$$, the language model learns to predict the probability of next token given the history.

In the forward pass, the history contains words before the target token,

$$
p(x_1, \dots, x_n) = \prod_{i=1}^n p(x_i \mid x_1, \dots, x_{i-1})
$$

In the backward pass, the history contains words after the target token,

$$
p(x_1, \dots, x_n) = \prod_{i=1}^n p(x_i \mid x_{i+1}, \dots, x_n)
$$

The predictions in both directions are modeled by multi-layer LSTMs with hidden states $$\overrightarrow{\mathbf{h}}_{i,\ell}$$ and $$\overleftarrow{\mathbf{h}}_{i,\ell}$$ for input token $$x_i$$ at the layer level $$\ell=1,\dots,L$$.
The final layer’s hidden state $$\mathbf{h}_{i,L} = [\overrightarrow{\mathbf{h}}_{i,L}; \overleftarrow{\mathbf{h}}_{i,L}]$$ is used to output the probabilities over tokens after softmax normalization. They share the embedding layer and the softmax layer, parameterized by $$\Theta_e$$ and $$\Theta_s$$ respectively.



![ELMo biLSTM]({{ '/assets/images/ELMo-biLSTM.png' | relative_url }})
{: style="width: 85%;" class="center"}
*Fig. 3. The biLSTM base model of ELMo. (Image source: recreated based on the figure in ["Neural Networks, Types, and Functional Programming"](http://colah.github.io/posts/2015-09-NN-Types-FP/) by <NAME>.)*

The model is trained to minimize the negative log likelihood (= maximize the log likelihood for true words) in both directions:

$$
\begin{aligned}
\mathcal{L} = - \sum_{i=1}^n \Big( 
\log p(x_i \mid x_1, \dots, x_{i-1}; \Theta_e, \overrightarrow{\Theta}_\text{LSTM}, \Theta_s) + \\
\log p(x_i \mid x_{i+1}, \dots, x_n; \Theta_e, \overleftarrow{\Theta}_\text{LSTM}, \Theta_s) \Big)
\end{aligned}
$$


### ELMo Representations

On top of a $$L$$-layer biLM, ELMo stacks all the hidden states across layers together by learning a task-specific linear combination. The hidden state representation for the token $$x_i$$ contains $$2L+1$$ vectors:

$$
R_i = \{ \mathbf{h}_{i,\ell} \mid \ell = 0, \dots, L \}
$$
where $$\mathbf{h}_{0, \ell}$$ is the embedding layer output and $$\mathbf{h}_{i, \ell} = [\overrightarrow{\mathbf{h}}_{i,\ell}; \overleftarrow{\mathbf{h}}_{i,\ell}]$$.
{'fraction_non_alphanumeric': 0.13188269729938698, 'fraction_numerical': 0.018335450378306733, 'mean_word_length': 4.717713924850016, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 12, 'https://': 17, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2759872', 'n_tokens_mistral': 5869, 'n_tokens_neox': 5332, 'n_words': 2175}