<jupyter_start><jupyter_text>Representing Text Text as TensorsIf we want to solve NLP tasks with Neural Networks, we need some way to represent text as tensors. Computers already represent textual characters as numbers that map to fonts on your screen using coding encodings such as ASCII or UTF-8. ![Ascii Code](../images/ASCII.png)We understand what each letter **represents**, and how all characters come together to form the words of a sentence. However, computers by themselves do not have such an understanding, and neural network has to learn the meaning during training.Therefore, we can use different approaches when representing text:* **Character-level representation**, when we represent text by treating each character as a number. Given that we have $C$ different characters in our text corpus, the word *Hello* would be represented by $5\times C$ tensor. Each letter would correspond to a tensor column in one-hot encoding.* **Word-level representation**, in which we create a **vocabulary** of all words in our text, and then represent words using one-hot encoding. This approach is somehow better, because each letter by itself does not have much meaning, and thus by using higher-level semantic concepts - words - we simplify the task for the neural network. However, given large dictionary size, we need to deal with high-dimensional sparse tensors.  Text Classification TaskIn this module, we will start with a simple text classification task based on **AG_NEWS** dataset, which is to classify news headlines into one of 4 categories: World, Sports, Business and Sci/Tech. This dataset is built into [`torchtext`](https://github.com/pytorch/text) module, so we can easily access it.<jupyter_code>import torch
import torchtext
import os
import collections
os.makedirs('./data',exist_ok=True)
train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')
classes = ['World', 'Sports', 'Business', 'Sci/Tech']<jupyter_output><empty_output><jupyter_text>Here, `train_dataset` and `test_dataset` contain iterators that return pairs of label (number of class) and text respectively, for example:<jupyter_code>next(train_dataset)<jupyter_output><empty_output><jupyter_text>So, let's print out the first 10 new headlines from our dataset:<jupyter_code>for i,x in zip(range(5),train_dataset):
    print(f"**{classes[x[0]]}** -> {x[1]}")<jupyter_output>**Sci/Tech** -> Carlyle Looks Toward Commercial Aerospace (Reuters) Reuters - Private investment firm Carlyle Group,\which has a reputation for making well-timed and occasionally\controversial plays in the defense industry, has quietly placed\its bets on another part of the market.
**Sci/Tech** -> Oil and Economy Cloud Stocks' Outlook (Reuters) Reuters - Soaring crude prices plus worries\about the economy and the outlook for earnings are expected to\hang over the stock market next week during the depth of the\summer doldrums.
**Sci/Tech** -> Iraq Halts Oil Exports from Main Southern Pipeline (Reuters) Reuters - Authorities have halted oil export\flows from the main pipeline in southern Iraq after\intelligence showed a rebel militia could strike\infrastructure, an oil official said on Saturday.
**Sci/Tech** -> Oil prices soar to all-time record, posing new menace to US economy (AFP) AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace ba[...]<jupyter_text>Because datasets are iterators, if we want to use the data multiple times we need to convert it to list:<jupyter_code>train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')
train_dataset = list(train_dataset)
test_dataset = list(test_dataset)<jupyter_output><empty_output><jupyter_text>Now we need to convert text into **numbers** that can be represented as tensors. If we want word-level representation, we need to do two things:* use **tokenizer** to split text into **tokens*** build a **vocabulary** of those tokens.<jupyter_code>tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
tokenizer('He said: hello')
counter = collections.Counter()
for (label, line) in train_dataset:
    counter.update(tokenizer(line))
vocab = torchtext.vocab.Vocab(counter, min_freq=1)<jupyter_output><empty_output><jupyter_text>Using vocabulary, we can easily encode out tokenied string into a set of numbers:<jupyter_code>vocab_size = len(vocab)
print(f"Vocab size if {vocab_size}")

def encode(x):
    return [vocab.stoi[s] for s in tokenizer(x)]

encode('I love to play with my words')<jupyter_output>Vocab size if 95812<jupyter_text>Bag of Words Text RepresentationBecause words represent meaning, sometimes we can figure out the meaning of a text by just looking at the invididual words, regardless of their order in the sentence. For example, when classifying news, words like *weather*, *snow* are likely to indicate *weather forecast*, while words like *stocks*, *dollar* would count towards *financial news*.**Bag of Words** (BoW) vector representation is the most commonly used traditional vector representation. Each word is linked to a vector index, vector element contains the number of occurences of a word in a given document.![bow image here](../images/bow.png) > **Note**: You can also think of BoW as a sum of all one-hot-encoded vectors for individual words in the text.Below is an example of how to generate a bag of word representation using the Scikit Learn python library:<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
corpus = [
        'I like hot dogs.',
        'The dog ran fast.',
        'Its hot outside.',
    ]
vectorizer.fit_transform(corpus)
vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()<jupyter_output><empty_output><jupyter_text>To compute bag-of-words vector from the vector representation of our AG_NEWS dataset, we can use the following function:<jupyter_code>vocab_size = len(vocab)

def to_bow(text,bow_vocab_size=vocab_size):
    res = torch.zeros(bow_vocab_size,dtype=torch.float32)
    for i in encode(text):
        if i<bow_vocab_size:
            res[i] += 1
    return res

print(to_bow(train_dataset[0][1]))<jupyter_output>tensor([0., 0., 2.,  ..., 0., 0., 0.])<jupyter_text>> **Note:** Here we are using global `vocab_size` variable to specify default size of the vocabulary. Since often vocabulary size is pretty big, we can limit the size of the vocabulary to most frequent words. Try lowering `vocab_size` value and running the code below, and see how it affects the accuracy. You should expect some accuracy drop, but not dramatic, in lieu of higher performance.  Training BoW ClassifierNow that we have learnt how to build Bag-of-Words representation of our text, let's train a classifier on top of it. First, we need to convert our dataset for training in such a way, that all positional vector representations are converted to bag-of-words representation. This can be achieved by passing `bowify` function as `collate_fn` parameter to standard torch `DataLoader`:<jupyter_code>from torch.utils.data import DataLoader
import numpy as np 

# this collate function gets list of batch_size tuples, and needs to 
# return a pair of label-feature tensors for the whole minibatch
def bowify(b):
    return (
            torch.LongTensor([t[0]-1 for t in b]),
            torch.stack([to_bow(t[1]) for t in b])
    )

train_loader = DataLoader(train_dataset, batch_size=16, collate_fn=bowify, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=16, collate_fn=bowify, shuffle=True)<jupyter_output><empty_output><jupyter_text>Now let's define a simple classifier neural network that contains one linear layer. The size of the input vector equals to `vocab_size`, and output size corresponds to the number of classes (4). Because we are solving classification task, the final activation function is `LogSoftmax()`.<jupyter_code>net = torch.nn.Sequential(torch.nn.Linear(vocab_size,4),torch.nn.LogSoftmax(dim=1))<jupyter_output><empty_output><jupyter_text>Now we will define standard PyTorch training loop. Because our dataset is quite large, for our teachning purpose we will train only for one epoch, and sometimes even for less than an epoch (speficying `epoch_size` parameter allows us to limit training). We would also report accumulated training accuracy during training; the frequency of reporting is specified using `report_freq` parameter.<jupyter_code>def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.NLLLoss(),epoch_size=None, report_freq=200):
    optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
    net.train()
    total_loss,acc,count,i = 0,0,0,0
    for labels,features in dataloader:
        optimizer.zero_grad()
        out = net(features)
        loss = loss_fn(out,labels) #cross_entropy(out,labels)
        loss.backward()
        optimizer.step()
        total_loss+=loss
        _,predicted = torch.max(out,1)
        acc+=(predicted==labels).sum()
        count+=len(labels)
        i+=1
        if i%report_freq==0:
            print(f"{count}: acc={acc.item()/count}")
        if epoch_size and count>epoch_size:
            break
    return total_loss.item()/count, acc.item()/count
train_epoch(net,train_loader,epoch_size=15000)<jupyter_output>3200: acc=0.80875
6400: acc=0.845625
9600: acc=0.8625
12800: acc=0.86875<jupyter_text>BiGrams, TriGrams and N-GramsOne limitation of a bag of words approach is that some words are part of multi word expresssions, for example, the word 'hot dog' has a completely different meaning than the words 'hot' and 'dog' in other contexts. If we represent words 'hot` and 'dog' always by the same vectors, it can confuse our model.To address this, **N-gram representations** are often used in methods of document classification, where the frequency of each word, bi-word or tri-word is a useful feature for training classifiers. In bigram representation, for example, we will add all word pairs to the vocabulary, in addition to original words. Below is an example of how to generate a bigram bag of word representation using the Scikit Learn:<jupyter_code>bigram_vectorizer = CountVectorizer(ngram_range=(1, 2), token_pattern=r'\b\w+\b', min_df=1)
corpus = [
        'I like hot dogs.',
        'The dog ran fast.',
        'Its hot outside.',
    ]
bigram_vectorizer.fit_transform(corpus)
print("Vocabulary:\n",bigram_vectorizer.vocabulary_)
bigram_vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()<jupyter_output>Vocabulary:
 {'i': 7, 'like': 11, 'hot': 4, 'dogs': 2, 'i like': 8, 'like hot': 12, 'hot dogs': 5, 'the': 16, 'dog': 0, 'ran': 14, 'fast': 3, 'the dog': 17, 'dog ran': 1, 'ran fast': 15, 'its': 9, 'outside': 13, 'its hot': 10, 'hot outside': 6}<jupyter_text>The main drawback of N-gram approach is that vocabulary size starts to grow extremely fast. In practice, we need to combine N-gram representation with some dimensionality reduction techniques, such as *embeddings*, which we will discuss in the next unit.To use N-gram representation in our **AG News** dataset, we need to build special ngram vocabulary:<jupyter_code>counter = collections.Counter()
for (label, line) in train_dataset:
    l = tokenizer(line)
    counter.update(torchtext.data.utils.ngrams_iterator(l,ngrams=2))
    
bi_vocab = torchtext.vocab.Vocab(counter, min_freq=1)

print("Bigram vocabulary length = ",len(bi_vocab))<jupyter_output>Bigram vocabulary length =  1308844<jupyter_text>We could then use the same code as above to train the classifier, however, it would be very memory-inefficient. In the next unit, we will train bigram classifier using embeddings.> **Note:** You can only leave those ngrams that occur in the text more than specified number of times. This will make sure that infrequent bigrams will be omited, and will decrease the dimensionality significanlty. To do this, set `min_freq` parameter to a higher value, and observe the length of vocabulary change.  Term Frequency Inverse Document Frequency TF-IDFIn BoW representation, word occurrences are evenly weighted, regardless of the word itself. However, it is clear that frequent words, such as *a*, *in*, etc. are much less important for the classification, than specialized terms. In fact, in most NLP tasks some words are more relevant than others.**TF-IDF** stands for **term frequency–inverse document frequency**. It is a variation of bag of words, where instead of a binary 0/1 value indicating the appearence of a word in a document, a floating-point value is used, which is related to the frequency of word occurence in the corpus.More formally, the weight $w_{ij}$ of a word $i$ in the document $j$ is defined as:$$w_{ij} = tf_{ij}\times\log({N\over df_i})$$where* $tf_{ij}$ is the number of occurences of $i$ in $j$, i.e. the BoW value we have seen before* $N$ is the number of documents in the collection* $df_i$ is the number of documents containing the word $i$ in the whole collectionTF-IDF value $w_{ij}$ increases proportionally to the number of times a word appears in a document and is offset by the number of documents in the corpus that contains the word, which helps to adjust for the fact that some words appear more frequently than others. For example, if the word appears in *every* document in the collection, $df_i=N$, and $w_{ij}=0$, and those terms would be completely disregarded.You can easily create TF-IDF vectorization of text using Scikit Learn:<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
vectorizer = TfidfVectorizer(ngram_range=(1,2))
vectorizer.fit_transform(corpus)
vectorizer.transform(['My dog likes hot dogs on a hot day.']).toarray()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08706776811170407, 'fraction_numerical': 0.010015351999415162, 'mean_word_length': 5.07190412782956, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 60, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '151375', 'license': 'MIT', 'n_tokens_mistral': 4022, 'n_tokens_neox': 3832, 'n_words': 1667, 'path': 'nlp-pytorch/2-representing-text.ipynb', 'repo_name': 'aiadvocates/pytorchfundamentals', 'size': '13679'}