<commit_before>---
layout: article
title: NLP
comments: true
categories: data_science
image:
  teaser: jupyter-main-logo.svg
---

## Representation
- Natural language is the means of communication for humans. It represents concepts, entities, and the relationships among them in the real word and therefore is quite complex. However, on the representation level it consists of discrete units of letters, words, and sentences. 

-  Mathematically speaking, a language model is the joint probability distribution of words such that correct and meaningful sequence of words/characters have a high probability distribution while incorrect sequences have low probability. This joint distribution can be factorized further based on assumptions like n-gram models meaning that only n sequences depend on each other and not more. 

- In order to be able to perform machine learning on natural language, we have to first represent it in a numerical format. The central problem in NLP seems to be representation of natural language as numbers. After that the problem is simply sequence and structure learning where a vast array of tools from machine learning exist. 

- the process of turning language symbols into sth we can work with is called tokenization. 
    + Manual tokenization (example my TextRep class)
    + Many of the best practices for tokenizing raw text have been captured and made available in a Python library called the Natural Language Toolkit or NLTK for short.

- Basic representation started by defining a dictionary as the list of all allowed words or letter combinations. Common representations are:
    + Ascii code representation: Doesn't tell you anything about the meaning of the word. 
        * Distance between words are not well defined.
    + tf-idf score: tf is a normalized term frequency in the document while idf is a weight that corrects for more frequent words like 'the'. The tf-idf score is the product $$tf.idf$$$.
        * tf-idf is a measure of the relative importance of a word in a document. 
        * It doesn't bear any information about semantic relationships of words.
    + one-hot encoding: a vector with the size of the dictionary where all entries are zero except the single entry corresponding to the word in a location. 
        * The distance between all words is a constant one without encoding relationships between words, no notion of meaning is present, huge dimensionality problem.
    + Bag of words (BoW): add up one-hot encodings of words in each document (word counts) and normalize. 
        * Latent semantic analysis (LSA) models would do matrix factorization (SVD) on the bag of words representations or bigram word frequency matrix. They pick the eigen-vectors as word representations which can encode some semantic and syntactic (part of speech) information. 
        * The same linear relationships between word vectors in word2vec were also observed in LSA to a lower extent with proper scaling of bag of words frequencies. 
            - scaling involves: weighing the co-occurrence count based on distance between the words in the document, ignoring function words like "he", "she", etc.
- Word2Vec models: a class of NN models that learn a vector representation for words from an unlabeled corpus of text. 
    + Skip-Gram: Based on the notion that words appearing in similar contexts are related to each other semantically. Skip-gram predicts a window of neighboring words from a single word.
        * We convert the corpus into samples with a neighboring window of n-grams on both sides of a word.
        * input is one-hot representation of a single words. each neighbor word has a corresponding output. Therefore, outputs are n words in the neighborhood (both sides), each having a one-hot representation.
        * The model has a bottleneck hidden layer between the input word and output neighboring words. The weights between one-hot input and hidden layer will be used as m-dimensional word embedding matrix (W) after training. 
        * no nonlinearity is used in the bottleneck layer since inputs are one-hot and only activate a single row of the weight matrix.
        * the weight matrix between the bottleneck layer and each output node is shared.
        *  a softmax is used for each neighboring word output to produce a one-hot vector for that neighboring word.
            -  In the two-class logistic regression, we pass the single output through a sigmoid function, $$\frac{1}{1+exp(-x)}$$, to crush it into the $$[0,1]$$ range and interpret it as a probability i.e. $$P(Y=0)= sigmoid(x), P(Y=1)=1-P(Y=0)$$. 
            -  In the multi-class logistic regression, with have $$K$$ outputs so we can't just say one minus probability of one class. Therefore, we need a generalization of the sigmoid to crush all the outputs into the $$[0,1]$$ range to interpret them as probabilities. If we write the output equations for the other class in the two class logistic regression, we can [derive the generalization](https://stats.stackexchange.com/questions/233658/softmax-vs-sigmoid-function-in-logistic-classifier) to softmax function $$P(Y=j)=\frac{exp(z_j)}{\sum_k exp(z_k)}$$.
        * The loss function is negative log-likelihood loss i.e. $$L= -\frac{1}{n} \sum_k \log(a)$$
            - In two class logistic regression, we can use cross-entropy loss with the sigmoid output since there are only two outputs i.e. $$L= - \frac{1}{n} \sum_x (y \log(o) + (1-y)\log(1-o))$$

            - In the multiclass logistic regression, the equivalent of cross-entropy for softmax output is negative log-likelihood loss i.e. $$L= - \frac{1}{n} \sum_x \log(o_y)$$.

            - The problem is using softmax with thousands of outputs (one-hot words) as a multi-class classification is very costly. So the idea of training word-vectors with negative sampling comes from density ratio estimation using noise contrastive estimation (NCE). The basic idea is to convert the huge-muliclass classification problem (i.e. predict a distribution over all words) to a density ratio problem that can be solved with a binary classification of distinguishing true sequences of words (data) from fake sequences of words (keeping central word and randomly sampling the context). 
                + The negative samples are chosen from a slightly modified distribution that favors rare words.

        * Training Algo for word embeddings: 
            - Represent each word as a d dimensional vector (or connect one-hot word input to a d-dim bottleneck so that weight matrix (W) becomes a d-dim word representation).
            - Represent each context word as a d dimensional vector (or connect the d-dim bottleneck layer to n one-hot outputs for contexts).
            - Initalize all vectors to random weights.
            - Arrange vectors in two matrices, W and C.
            - feed in the corpus formatted as bi-ngrams. 
            - Negative sampling objective is based on NCE which basically constructs a fake dataset of bi-ngrams from the original bi-ngrams by replacing the input word with a random one while keeping the context the same. Objective then tries to assign high probability to correct bi-ngrams while assigning low probability to fake bi-ngrams.
            - After training, Throw away the C matrix and use the W matrix as word-embeddings. 
        * If we don't throw away the C matrix, the product $$W.C$$ will produce the co-occurrence frequency matrix for each word and its ngrams. Therefore, word2vec is essentially doing a matrix factorization on that matrix in a more algorithmically efficient way than LSA. Even LDA can also be interpreted as a matrix factorization.
       
    + Continuous BoW (CBoW): It's the exact mirror of skip-gram model where a word is predicted from it's neighbors instead. We train a simple one-layer logistic regression model to predict the one-hot encoding of a word from the one-hot representation of its neighbors. 
        * After the training with maximum likelihood, the weight matrix can be used as a word embeddings. 
        * typically used as a pre-training for initialization of the word-vectors before learning an embedding. 
    + word2Vec encoding: Based on the notion that words appearing in similar contexts are related to each other semantically. We learn word vector representations by defining a model that predicts a word given its context and context given a word. We condition a word on its neighbors. The probability of a word (center) given its neighbors (n) is determined by the normalized softmax of the distance between two word vectors (dot product distance for skip-gram) $$p(n|c)=\frac{\exp{u_n^T . u_c}}{\sum_c \exp{u_n^T . u_c}}$$ (i.e. angle between the vectors). 
        * We let the word embeddings be parameters. Training an LSTM with softmax to predict context from a word learns word2vec representations.
        * After learning the word embeddings, word vectors will be stored in a lookup table with an index for each word.
        * The similarity of two words is represented by the angle between their vectors (cosine similarity). Therefore, similar words will be parallel and very dissimilar words will be orthogonal. In one-hot all words are orthogonal which doesn't make sense given their semantic similarities.
        * the result is a dense vector representation of words that embeds words appearing in each others context in the same region of the vector space (low distance).
        * There usually is a linear relationship between word vectors for example relationship between king/queen is similar to man/woman
    + Glove: goes from word-counts (bag-of-words) to nice meaningful embedding properties of the word2vec model. The crucial insight is that co-occurrence probabilities of words might be volatile but the ratio of co-occurrence of words is a much more stable measure for embedding meaning. 
        * the way this ratio is mapped to word vectors is equating the probability with distance $$w_i.w_j=\log p(i|j)$$.
        * can we use density ratio estimation here?
    + Skip Thought Vectors: Generate sentence codes in the style of word embeddings to predict context sentences. One encoder and two decoder to predict previous and next sentences.
        * sentence_t -> sentence_{t-1} sentence_{t+1} [Kiros et al.,2015]
    + Paragraph vector: A paragraph is represented as a vector in the same space as the single-word embeddings.
        * non-RNN model. A paragraph matrix D is a bag-of-sentences representation and a vocabulary matrix W is a bag-of-words representation. we train on the bag-of-words W to predict next words using bag-of-sentences D as context.
    + Word-vector domain adaptation: Adapting word-vectors from a domain with large data to one with less data is a useful thing. However, each domain usually has it's own specific set of terms which makes using of general purpose embeddings challenging. Is it possible adapt general-purpose word embeddings to a specific domain?
        * The idea is to train word-vectors first on the larger domain, then use a regularizer on the word2vec objective that penalizes the L2 distance between the embeddings of the shared words in the source and target domains. (see Yang et al 2017)
        * These distances are weighted by a significance function based on the notion that words that are similar in frequency of occurence are probably semantically not domain-specific (similar to Glove objective) and are very transferable (should have similar embeddings). 
        * Words that have different frequencies of occurance are domain-specific and are less transferable (less similar embeddings).
    + cross-lingual word embeddings: The idea is to align embedding spaces of words instead of lexicons. (Connea et al 2017) did unsupervised word alignment:
        * First two sets of word embeddings are trained independently on the two languages (X, Y). Model learns a mapping between them such that translations are close in the common space.
        * A GAN is used similar to cyclaGAN? The generator is (W.X) and the discriminator tells languages apart. They first train discriminator and then the generator.
        * Two extra steps are: 
            - remove noise that rare words introduce 
            - build translations using the learned mapping and a distance measure

- LDA: While word2vec tries to predict a word from its local context, LDA tries to predict the word from the global document (LDA generative model first chooses a topic, then samples a word from that topic). While word2vec representations are dense, LDA representations are sparse (only a few topics).

- Obviously the number of words in a language is much smaller than the combination of the letters in that language. Why not learn a language-specific word manifold that maps characters to words? Then we can use the char-level language models instead of word-level models and reduce the dimensionality of the problem since number of chars are multiple orders of magnitude smaller than words. 

- a measure: can we take these word vectors and use them in other task like language modeling, translation, etc


### Batching variable length sequences:
- There are two ways of batching variable length sequences:
    + Packing sequences of same size together in a minibatch and sending that into the LSTM, but that's not always possible. 
    + Padding sequnces with zero so that all have the same maximum seq-len size. This can be done two ways:
        * Simply feeding the padded sequences in a minibatch and get a fixed length of output. The desired output should have different lengths since sequences had different lengths and RNN should have unrolled only for the real sequences, so we have to mask them manually.
            - mask = (time < length).float().unsqueeze(1).expand_as(h_next)
            - h_next = h_next*mask + hx[0]*(1 - mask)
            - c_next = c_next*mask + hx[1]*(1 - mask)
        * Using pytorch "pack_padded_sequence" which does a combination of packing and masking so that the output of each example will have different length. The RNN output then has to be unpacked using "pad_packed_sequence". 
            - Pad variable length sequences in a batch with zeros 
            - Sort the minibatch so that the longest sequence is at the beginning
            - pass the tensor and the lengths of sequences to "pack_padded_sequence"
            - the output of "pack_padded_sequence" goes to the RNN
            - RNN output is passed to "pad_packed_sequence" to map the output back to a zero-padded tensor corresponding to the right seq sizes.

### Pytorch Dataset class:
Pytorch has a dataset class that provides some tools for easy loading of data. 
    - your dataset class should inherit **"torch.utils.data.dataset"** Several methods need to be implemented:
        + __init__(self) load and preprocess the data here or in __getitem__ for memory efficiency.
        + __len__(self) returns the size of the dataset.
        + __getitem__(self) indexes into dataset such that dataset[i] returns i-th sample.
    - In order to shuffle and minibatch the data, pytorch provides an iterator class i.e. **"torch.utils.data.DataLoader"**. We specify how exactly the samples need to be batched using **collate_fn** if we need to. Usage in training is as simple as instantiating the Dataloader class with the dataset instence and a for loop with **enumerate(dataloader)**.

### Language modeling:
A language model predicts the probability of next word given the history of words seen so far i.e. P(next word | history). Given a proposed probability model q, one may evaluate q by asking how well it predicts a separate test sample. 

The best known metrics for evaluating a language model is model perplexity on test data $$T = {w_1, ..., w_t}$$. Perplexity quantifies the average number of bits needed to represent a test event if one uses a code based on the model. It is defined as 2 or e to the power of the average model entropy (i.e. average probability assigned to each word in the test set). 

Cross entropy is the belief of the model about the likelihood of targets (real next words) under the model and shows the discrepency between reality and model beliefs. If we average the cross_entropy losses of all the words in the test set, we get average test set entropy under the model. Therefore exp(cross_entropy_loss) is perplexity. 



### NLP tasks
Easy
• Spell Checking
• Keyword Search
• Finding Synonyms
Medium
• Parsing information from websites, documents, etc.
Hard
• Machine Translation (e.g. Translate Chinese text to English)
• Semantic Analysis (What is the meaning of query statement?)
• Coreference (e.g. What does "he" or "it" refer to given a document?)
• Question Answering (e.g. Answering Jeopardy questions)


## Recurrent Weight dropout
- using nn.dropout(), a new binary dropout mask is sampled each and every time the dropout function is called even if the given connection is repeated (e.g. different input time steps recieve different dropout masks). 
- Basically what weight-drop does is to use the same dropout mask for all steps in both forward and backward directions(variational dropout). Applied to input-layer uses same mask for all inputs, and applied to a recurrent layer, uses the same dropout mask for recurrent steps. The important part is using same mask for recurrent steps that avoids drowning out the signal.
- For recurrent connections DropConnect (zeroing out individual weights as opposed to ) is recommended

### Embedding dropConnect
- zeroing out a percentage of weights of the embedding weight matrix helps with regularization of language models. 

### Cache pointer
After a word appears once in a document, it is much more likely to appear again. For example in a word like tiger with low frequency in the whole corpus of wikipedia is much more likely in an article about tigers. Cache pointer models have a cache component, which contain the words that appeared in the
recent history (e.g. a fixed number of words).

- Save the history (cache) of hidden activations and target words. Then perform a matrix vector product of the history matrix and the current hidden vector and use it as an attention score. 
- apply softmax to this attention score to get a distribution over the history
- apply this distribution to the history of target words to get relevant words that have actually appeared in text<commit_msg>unsupervised machine translation<commit_after>---
layout: article
title: NLP
comments: true
categories: data_science
image:
  teaser: jupyter-main-logo.svg
---

## Representation
- Natural language is the means of communication for humans. It represents concepts, entities, and the relationships among them in the real word and therefore is quite complex. However, on the representation level it consists of discrete units of letters, words, and sentences. 

-  Mathematically speaking, a language model is the joint probability distribution of words such that correct and meaningful sequence of words/characters have a high probability distribution while incorrect sequences have low probability. This joint distribution can be factorized further based on assumptions like n-gram models meaning that only n sequences depend on each other and not more. 

- In order to be able to perform machine learning on natural language, we have to first represent it in a numerical format. The central problem in NLP seems to be representation of natural language as numbers. After that the problem is simply sequence and structure learning where a vast array of tools from machine learning exist. 

- the process of turning language symbols into sth we can work with is called tokenization. 
    + Manual tokenization (example my TextRep class)
    + Many of the best practices for tokenizing raw text have been captured and made available in a Python library called the Natural Language Toolkit or NLTK for short.

- Basic representation started by defining a dictionary as the list of all allowed words or letter combinations. Common representations are:
    + Ascii code representation: Doesn't tell you anything about the meaning of the word. 
        * Distance between words are not well defined.
    + tf-idf score: tf is a normalized term frequency in the document while idf is a weight that corrects for more frequent words like 'the'. The tf-idf score is the product $$tf.idf$$$.
        * tf-idf is a measure of the relative importance of a word in a document. 
        * It doesn't bear any information about semantic relationships of words.
    + one-hot encoding: a vector with the size of the dictionary where all entries are zero except the single entry corresponding to the word in a location. 
        * The distance between all words is a constant one without encoding relationships between words, no notion of meaning is present, huge dimensionality problem.
    + Bag of words (BoW): add up one-hot encodings of words in each document (word counts) and normalize. 
        * Latent semantic analysis (LSA) models would do matrix factorization (SVD) on the bag of words representations or bigram word frequency matrix. They pick the eigen-vectors as word representations which can encode some semantic and syntactic (part of speech) information. 
        * The same linear relationships between word vectors in word2vec were also observed in LSA to a lower extent with proper scaling of bag of words frequencies. 
            - scaling involves: weighing the co-occurrence count based on distance between the words in the document, ignoring function words like "he", "she", etc.
- Word2Vec models: a class of NN models that learn a vector representation for words from an unlabeled corpus of text. 
    + Skip-Gram: Based on the notion that words appearing in similar contexts are related to each other semantically. Skip-gram predicts a window of neighboring words from a single word.
        * We convert the corpus into samples with a neighboring window of n-grams on both sides of a word.
        * input is one-hot representation of a single words. each neighbor word has a corresponding output. Therefore, outputs are n words in the neighborhood (both sides), each having a one-hot representation.
        * The model has a bottleneck hidden layer between the input word and output neighboring words. The weights between one-hot input and hidden layer will be used as m-dimensional word embedding matrix (W) after training. 
        * no nonlinearity is used in the bottleneck layer since inputs are one-hot and only activate a single row of the weight matrix.
        * the weight matrix between the bottleneck layer and each output node is shared.
        *  a softmax is used for each neighboring word output to produce a one-hot vector for that neighboring word.
            -  In the two-class logistic regression, we pass the single output through a sigmoid function, $$\frac{1}{1+exp(-x)}$$, to crush it into the $$[0,1]$$ range and interpret it as a probability i.e. $$P(Y=0)= sigmoid(x), P(Y=1)=1-P(Y=0)$$. 
            -  In the multi-class logistic regression, with have $$K$$ outputs so we can't just say one minus probability of one class. Therefore, we need a generalization of the sigmoid to crush all the outputs into the $$[0,1]$$ range to interpret them as probabilities. If we write the output equations for the other class in the two class logistic regression, we can [derive the generalization](https://stats.stackexchange.com/questions/233658/softmax-vs-sigmoid-function-in-logistic-classifier) to softmax function $$P(Y=j)=\frac{exp(z_j)}{\sum_k exp(z_k)}$$.
        * The loss function is negative log-likelihood loss i.e. $$L= -\frac{1}{n} \sum_k \log(a)$$
            - In two class logistic regression, we can use cross-entropy loss with the sigmoid output since there are only two outputs i.e. $$L= - \frac{1}{n} \sum_x (y \log(o) + (1-y)\log(1-o))$$

            - In the multiclass logistic regression, the equivalent of cross-entropy for softmax output is negative log-likelihood loss i.e. $$L= - \frac{1}{n} \sum_x \log(o_y)$$.

            - The problem is using softmax with thousands of outputs (one-hot words) as a multi-class classification is very costly. So the idea of training word-vectors with negative sampling comes from density ratio estimation using noise contrastive estimation (NCE). The basic idea is to convert the huge-muliclass classification problem (i.e. predict a distribution over all words) to a density ratio problem that can be solved with a binary classification of distinguishing true sequences of words (data) from fake sequences of words (keeping central word and randomly sampling the context). 
                + The negative samples are chosen from a slightly modified distribution that favors rare words.

        * Training Algo for word embeddings: 
            - Represent each word as a d dimensional vector (or connect one-hot word input to a d-dim bottleneck so that weight matrix (W) becomes a d-dim word representation).
            - Represent each context word as a d dimensional vector (or connect the d-dim bottleneck layer to n one-hot outputs for contexts).
            - Initalize all vectors to random weights.
            - Arrange vectors in two matrices, W and C.
            - feed in the corpus formatted as bi-ngrams. 
            - Negative sampling objective is based on NCE which basically constructs a fake dataset of bi-ngrams from the original bi-ngrams by replacing the input word with a random one while keeping the context the same. Objective then tries to assign high probability to correct bi-ngrams while assigning low probability to fake bi-ngrams.
            - After training, Throw away the C matrix and use the W matrix as word-embeddings. 
        * If we don't throw away the C matrix, the product $$W.C$$ will produce the co-occurrence frequency matrix for each word and its ngrams. Therefore, word2vec is essentially doing a matrix factorization on that matrix in a more algorithmically efficient way than LSA. Even LDA can also be interpreted as a matrix factorization.
       
    + Continuous BoW (CBoW): It's the exact mirror of skip-gram model where a word is predicted from it's neighbors instead. We train a simple one-layer logistic regression model to predict the one-hot encoding of a word from the one-hot representation of its neighbors. 
        * After the training with maximum likelihood, the weight matrix can be used as a word embeddings. 
        * typically used as a pre-training for initialization of the word-vectors before learning an embedding. 
    + word2Vec encoding: Based on the notion that words appearing in similar contexts are related to each other semantically. We learn word vector representations by defining a model that predicts a word given its context and context given a word. We condition a word on its neighbors. The probability of a word (center) given its neighbors (n) is determined by the normalized softmax of the distance between two word vectors (dot product distance for skip-gram) $$p(n|c)=\frac{\exp{u_n^T . u_c}}{\sum_c \exp{u_n^T . u_c}}$$ (i.e. angle between the vectors). 
        * We let the word embeddings be parameters. Training an LSTM with softmax to predict context from a word learns word2vec representations.
        * After learning the word embeddings, word vectors will be stored in a lookup table with an index for each word.
        * The similarity of two words is represented by the angle between their vectors (cosine similarity). Therefore, similar words will be parallel and very dissimilar words will be orthogonal. In one-hot all words are orthogonal which doesn't make sense given their semantic similarities.
        * the result is a dense vector representation of words that embeds words appearing in each others context in the same region of the vector space (low distance).
        * There usually is a linear relationship between word vectors for example relationship between king/queen is similar to man/woman
    + Glove: goes from word-counts (bag-of-words) to nice meaningful embedding properties of the word2vec model. The crucial insight is that co-occurrence probabilities of words might be volatile but the ratio of co-occurrence of words is a much more stable measure for embedding meaning. 
        * the way this ratio is mapped to word vectors is equating the probability with distance $$w_i.w_j=\log p(i|j)$$.
        * can we use density ratio estimation here?
    + Skip Thought Vectors: Generate sentence codes in the style of word embeddings to predict context sentences. One encoder and two decoder to predict previous and next sentences.
        * sentence_t -> sentence_{t-1} sentence_{t+1} [Kiros et al.,2015]
    + Paragraph vector: A paragraph is represented as a vector in the same space as the single-word embeddings.
        * non-RNN model. A paragraph matrix D is a bag-of-sentences representation and a vocabulary matrix W is a bag-of-words representation. we train on the bag-of-words W to predict next words using bag-of-sentences D as context.
    + Word-vector domain adaptation: Adapting word-vectors from a domain with large data to one with less data is a useful thing. However, each domain usually has it's own specific set of terms which makes using of general purpose embeddings challenging. Is it possible adapt general-purpose word embeddings to a specific domain?
        * The idea is to train word-vectors first on the larger domain, then use a regularizer on the word2vec objective that penalizes the L2 distance between the embeddings of the shared words in the source and target domains. (see Yang et al 2017)
        * These distances are weighted by a significance function based on the notion that words that are similar in frequency of occurence are probably semantically not domain-specific (similar to Glove objective) and are very transferable (should have similar embeddings). 
        * Words that have different frequencies of occurance are domain-specific and are less transferable (less similar embeddings).
    + cross-lingual word embeddings: The idea is to align embedding spaces of words instead of lexicons. (Connea et al 2017) did unsupervised word alignment:
        * First two sets of word embeddings are trained independently on the two languages (X, Y). Model learns a mapping between them such that translations are close in the common space.
        * A GAN is used similar to cyclaGAN? The generator is (W.X) and the discriminator tells languages apart. They first train discriminator and then the generator.
        * Two extra steps are: 
            - remove noise that rare words introduce 
            - build translations using the learned mapping and a distance measure

- LDA: While word2vec tries to predict a word from its local context, LDA tries to predict the word from the global document (LDA generative model first chooses a topic, then samples a word from that topic). While word2vec representations are dense, LDA representations are sparse (only a few topics).

- Obviously the number of words in a language is much smaller than the combination of the letters in that language. Why not learn a language-specific word manifold that maps characters to words? Then we can use the char-level language models instead of word-level models and reduce the dimensionality of the problem since number of chars are multiple orders of magnitude smaller than words. 

- a measure: can we take these word vectors and use them in other task like language modeling, translation, etc


### Batching variable length sequences:
- There are two ways of batching variable length sequences:
    + Packing sequences of same size together in a minibatch and sending that into the LSTM, but that's not always possible. 
    + Padding sequnces with zero so that all have the same maximum seq-len size. This can be done two ways:
        * Simply feeding the padded sequences in a minibatch and get a fixed length of output. The desired output should have different lengths since sequences had different lengths and RNN should have unrolled only for the real sequences, so we have to mask them manually.
            - mask = (time < length).float().unsqueeze(1).expand_as(h_next)
            - h_next = h_next*mask + hx[0]*(1 - mask)
            - c_next = c_next*mask + hx[1]*(1 - mask)
        * Using pytorch "pack_padded_sequence" which does a combination of packing and masking so that the output of each example will have different length. The RNN output then has to be unpacked using "pad_packed_sequence". 
            - Pad variable length sequences in a batch with zeros 
            - Sort the minibatch so that the longest sequence is at the beginning
            - pass the tensor and the lengths of sequences to "pack_padded_sequence"
            - the output of "pack_padded_sequence" goes to the RNN
            - RNN output is passed to "pad_packed_sequence" to map the output back to a zero-padded tensor corresponding to the right seq sizes.

### Pytorch Dataset class:
Pytorch has a dataset class that provides some tools for easy loading of data. 
    - your dataset class should inherit **"torch.utils.data.dataset"** Several methods need to be implemented:
        + __init__(self) load and preprocess the data here or in __getitem__ for memory efficiency.
        + __len__(self) returns the size of the dataset.
        + __getitem__(self) indexes into dataset such that dataset[i] returns i-th sample.
    - In order to shuffle and minibatch the data, pytorch provides an iterator class i.e. **"torch.utils.data.DataLoader"**. We specify how exactly the samples need to be batched using **collate_fn** if we need to. Usage in training is as simple as instantiating the Dataloader class with the dataset instence and a for loop with **enumerate(dataloader)**.

### Language modeling:
A language model predicts the probability of next word given the history of words seen so far i.e. P(next word | history). Given a proposed probability model q, one may evaluate q by asking how well it predicts a separate test sample. 

The best known metrics for evaluating a language model is model perplexity on test data $$T = {w_1, ..., w_t}$$. Perplexity quantifies the average number of bits needed to represent a test event if one uses a code based on the model. It is defined as 2 or e to the power of the average model entropy (i.e. average probability assigned to each word in the test set). 

Cross entropy is the belief of the model about the likelihood of targets (real next words) under the model and shows the discrepency between reality and model beliefs. If we average the cross_entropy losses of all the words in the test set, we get average test set entropy under the model. Therefore exp(cross_entropy_loss) is perplexity. 




### NLP tasks
Easy
• Spell Checking
• Keyword Search
• Finding Synonyms
Medium
• Parsing information from websites, documents, etc.
Hard
• Machine Translation (e.g. Translate Chinese text to English)
• Semantic Analysis (What is the meaning of query statement?)
• Coreference (e.g. What does "he" or "it" refer to given a document?)
• Question Answering (e.g. Answering Jeopardy questions)


## Unsupervised Machine Translation Using Monolingual Corpora Only 
The paper uses two unparalleled corpus (e.g. English novel, Spanish news), to perform translation. 

- Uses two denoising autoencoders (seq2seq) to reconstruct perturbed inputs from each corpus (autoencoder loss). (Noisifies input by randomly dropping a word or shift each word from its position with the constrained that a token can shift from its position at most k tokens to the left or to the right)

- Training the autoencoder to produce a sentence in corpus B from a sentence in corpus A and vice-versa. adding the BCE loss of the two together (translation loss). 

- Additionally a discriminator is used on the output of the encoder that discriminates which language the sentence is from (adversarial cost).

- The three terms of, autoencoder loss, translation loss , and discriminator loss are added up

Training loop:
1- Obtain translation using encoder of Language A and Decoder of Language B (translation)
2- Train each Autoencoder to regenerate an uncorrupted sentence when given a corrupted sentence (autoencoders)
3- corrupt the translation obtained in Step 1 , and recreating it. For this step the encoder of Language A , and Decoder of Language B are trained together (and also encoder of Language B and Decoder of Language A )

## Recurrent Weight dropout
- using nn.dropout(), a new binary dropout mask is sampled each and every time the dropout function is called even if the given connection is repeated (e.g. different input time steps recieve different dropout masks). 
- Basically what weight-drop does is to use the same dropout mask for all steps in both forward and backward directions(variational dropout). Applied to input-layer uses same mask for all inputs, and applied to a recurrent layer, uses the same dropout mask for recurrent steps. The important part is using same mask for recurrent steps that avoids drowning out the signal.
- For recurrent connections DropConnect (zeroing out individual weights as opposed to ) is recommended

### Embedding dropConnect
- zeroing out a percentage of weights of the embedding weight matrix helps with regularization of language models. 

### Cache pointer
After a word appears once in a document, it is much more likely to appear again. For example in a word like tiger with low frequency in the whole corpus of wikipedia is much more likely in an article about tigers. Cache pointer models have a cache component, which contain the words that appeared in the
recent history (e.g. a fixed number of words).

- Save the history (cache) of hidden activations and target words. Then perform a matrix vector product of the history matrix and the current hidden vector and use it as an attention score. 
- apply softmax to this attention score to get a distribution over the history
- apply this distribution to the history of target words to get relevant words that have actually appeared in text
{'fraction_non_alphanumeric': 0.04634599774816056, 'fraction_numerical': 0.002854075567542091, 'mean_word_length': 3.743758539311887, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6121227', 'n_tokens_mistral': 9157, 'n_tokens_neox': 8701, 'n_words': 5604}