<jupyter_start><jupyter_text><jupyter_code>!pip install -U torch==1.8.0 torchtext==0.9.0

# Reload environment
exit()<jupyter_output><empty_output><jupyter_text>This is a tutorial to show how to migrate from the legacy API in torchtext to the new API in 0.9.0 release. Here, we take the IMDB dataset as an example for the sentiment analysis. Both legacy and new APIs in torchtext can preprocess the text input and prepare the data to train/validate a model with the following steps:*   Train/validate/test split: generate train/validate/test data set if they are available*   Tokenization: break a raw text string sentence into a list of words*   Vocab: define a "contract" from tokens to indexes*   Numericalize: convert a list of tokens to the corresponding indexes*   Batch: generate batches of data samples and add padding if necessaryIt should be noted that all the legacy features are still available, but within torchtext.legacy instead of torchtext.  Step 1: Create a dataset object----------------------------Fist of all, we create a dataset for the sentiment analysis. The individual data sample contains a label and a text string. *Legacy*In the legacy code, `Field` class is used for data processing, including tokenizer and numberzation. To check out the dataset, users need to first set up the TEXT/LABEL fields.<jupyter_code>import torchtext
import torch
from torchtext.legacy import data
from torchtext.legacy import datasets

TEXT = data.Field()
LABEL = data.LabelField(dtype = torch.long)
legacy_train, legacy_test = datasets.IMDB.splits(TEXT, LABEL)  # datasets here refers to torchtext.legacy.datasets<jupyter_output><empty_output><jupyter_text>You can print out the raw data by checking out Dataset.examples. The entire text data are stored as a list of tokens.<jupyter_code>legacy_examples = legacy_train.examples
print(legacy_examples[0].text, legacy_examples[0].label)<jupyter_output><empty_output><jupyter_text>*New*The new dataset API returns the train/test dataset split directly without the preprocessing information. Each split is an iterator which yields the raw texts and labels line-by-line.<jupyter_code>from torchtext.datasets import IMDB
train_iter, test_iter = IMDB(split=('train', 'test'))<jupyter_output><empty_output><jupyter_text>To print out the raw data, you can call the next() function on the IterableDataset.<jupyter_code>next(train_iter)<jupyter_output><empty_output><jupyter_text>Step 2 Build the data processing pipeline---------------------------- *Legacy*The default tokenizer implemented in the `Field` class is the built-in python `split()` function. Users choose the tokenizer by calling `data.get_tokenizer()`, and add it to the `Field` constructor. For the sequence model, it's common to append `` (begin-of-sentence) and `` (end-of-sentence) tokens, and the special tokens need to be defined in the `Field` class.<jupyter_code>TEXT = data.Field(tokenize=data.get_tokenizer('basic_english'),
                  init_token='<SOS>', eos_token='<EOS>', lower=True)
LABEL = data.LabelField(dtype = torch.long)
legacy_train, legacy_test = datasets.IMDB.splits(TEXT, LABEL)  # datasets here refers to torchtext.legacy.datasets<jupyter_output><empty_output><jupyter_text>Now you can create a vocabulary of the words from the text file stored in the predefined `Field` object, `TEXT`. You fist have to build a vocabulary in your `Field` object by passing the dataset to the `build_vocab` func. The Field object builds the vocabulary (`TEXT.vocab`) on a specific data split.<jupyter_code>TEXT.build_vocab(legacy_train)
LABEL.build_vocab(legacy_train)<jupyter_output><empty_output><jupyter_text>Things you can do with a vocabuary object*   Total length of the vocabulary*   String2Index (stoi) and Index2String (itos)*   A purpose-specific vocabulary which contains word appearing more than N times<jupyter_code>legacy_vocab = TEXT.vocab
print("The length of the legacy vocab is", len(legacy_vocab))
legacy_stoi = legacy_vocab.stoi
print("The index of 'example' is", legacy_stoi['example'])
legacy_itos = legacy_vocab.itos
print("The token at index 686 is", legacy_itos[686])

# Set up the mim_freq value in the Vocab class
TEXT.build_vocab(legacy_train, min_freq=10)
legacy_vocab2 = TEXT.vocab
print("The length of the legacy vocab is", len(legacy_vocab2))<jupyter_output><empty_output><jupyter_text>*New*Users have the access to different kinds of tokenizers directly via `data.get_tokenizer()` function.<jupyter_code>from torchtext.data.utils import get_tokenizer
tokenizer = get_tokenizer('basic_english')<jupyter_output><empty_output><jupyter_text>To have more flexibility, users can build the vocabulary directly with the Vocab class. For example, the argument `min_freq` is to set up the cutoff frequency to in the vocabulary. The special tokens, like `` and `` can be assigned to the special symbols in the constructor of the Vocab class.<jupyter_code>from collections import Counter
from torchtext.vocab import Vocab

train_iter = IMDB(split='train')
counter = Counter()
for (label, line) in train_iter:
    counter.update(tokenizer(line))
vocab = Vocab(counter, min_freq=10, specials=('<unk>', '<BOS>', '<EOS>', '<PAD>'))
print("The length of the new vocab is", len(vocab))
new_stoi = vocab.stoi
print("The index of '<BOS>' is", new_stoi['<BOS>'])
new_itos = vocab.itos
print("The token at index 2 is", new_itos[2])<jupyter_output><empty_output><jupyter_text>Both `text_transform` and `label_transform` are the callable object, such as a lambda func here, to process the raw text and label data from the dataset iterators. Users can add the special symbols `` and `` to the sentence in `text_transform`.<jupyter_code>text_transform = lambda x: [vocab['<BOS>']] + [vocab[token] for token in tokenizer(x)] + [vocab['<EOS>']]
label_transform = lambda x: 1 if x == 'pos' else 0

# Print out the output of text_transform
print("input to the text_transform:", "here is an example")
print("output of the text_transform:", text_transform("here is an example"))<jupyter_output><empty_output><jupyter_text>Step 3: Generate batch iterator--------------------------------To train a model efficiently, it's recommended to build an iterator to generate data batch. *Legacy*The legacy `Iterator` class is used to batch the dataset and send to the target device, like CPU or GPU.<jupyter_code>import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
legacy_train, legacy_test = datasets.IMDB.splits(TEXT, LABEL)  # datasets here refers to torchtext.legacy.datasets
legacy_train_iterator, legacy_test_iterator = data.Iterator.splits(
    (legacy_train, legacy_test), batch_size=8, device = device)<jupyter_output><empty_output><jupyter_text>For a NLP workflow, it's also common to define an iterator and batch texts with similar lengths together. The legacy `BucketIterator` class in torchtext library minimizes the amount of padding needed.<jupyter_code>from torchtext.legacy.data import BucketIterator
legacy_train, legacy_test = datasets.IMDB.splits(TEXT, LABEL)
legacy_train_bucketiterator, legacy_test_bucketiterator = data.BucketIterator.splits(
    (legacy_train, legacy_test),
    sort_key=lambda x: len(x.text),
    batch_size=8, device = device)<jupyter_output><empty_output><jupyter_text>*New*`torch.utils.data.DataLoader` is used to generate data batch. Users could customize the data batch by defining a function with the `collate_fn` argument in the DataLoader. Here, in the `collate_batch` func, we process the raw text data and add padding to dynamically match the longest sentence in a batch.<jupyter_code>from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pad_sequence

def collate_batch(batch):
   label_list, text_list = [], []
   for (_label, _text) in batch:
        label_list.append(label_transform(_label))
        processed_text = torch.tensor(text_transform(_text))
        text_list.append(processed_text)
   return torch.tensor(label_list), pad_sequence(text_list, padding_value=3.0)

train_iter = IMDB(split='train')
train_dataloader = DataLoader(list(train_iter), batch_size=8, shuffle=True, 
                              collate_fn=collate_batch)<jupyter_output><empty_output><jupyter_text>To group the texts with similar length together, like introduced in the legacy `BucketIterator` class, first of all, we randomly create multiple "pools", and each of them has a size of `batch_size * 100`. Then, we sort the samples within the individual pool by length. This idea can be implemented succintly through `batch_sampler` argument of PyTorch `Dataloader`. `batch_sampler` accepts 'Sampler' or Iterable object that yields indices of next batch. In the code below, we implemented a generator that yields batch of indices for which the corresponding batch of data is of similar length.<jupyter_code>import random

train_iter = IMDB(split='train')
train_list = list(train_iter)
batch_size = 8  # A batch size of 8

def batch_sampler():
    indices = [(i, len(tokenizer(s[1]))) for i, s in enumerate(train_list)]
    random.shuffle(indices)
    pooled_indices = []
    # create pool of indices with similar lengths 
    for i in range(0, len(indices), batch_size * 100):
        pooled_indices.extend(sorted(indices[i:i + batch_size * 100], key=lambda x: x[1]))

    pooled_indices = [x[0] for x in pooled_indices]

    # yield indices for current batch
    for i in range(0, len(pooled_indices), batch_size):
        yield pooled_indices[i:i + batch_size]

bucket_dataloader = DataLoader(train_list, batch_sampler=batch_sampler(),
                               collate_fn=collate_batch)

print(next(iter(bucket_dataloader)))<jupyter_output><empty_output><jupyter_text>Step 4: Iterate batch to train a model-------------------------------It's almost same for both legacy and new APIs to iterate the data for batches during training and validating a model. *Legacy*The legacy batch iterator can be iterated or executed with `next()` method.<jupyter_code># for item in legacy_train_iterator:
#   model(item)

# Or
next(iter(legacy_train_iterator))<jupyter_output><empty_output><jupyter_text>*New*The batch iterator can be iterated or executed with `next()` method.<jupyter_code># for idx, (label, text) in enumerate(train_dataloader):
#   model(item)

# Or
next(iter(train_dataloader))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09801452784503632, 'fraction_numerical': 0.005230024213075061, 'mean_word_length': 5.331085223789087, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '292246', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3126, 'n_tokens_neox': 3052, 'n_words': 1089, 'path': 'examples/legacy_tutorial/migration_tutorial.ipynb', 'repo_name': 'Pandinosaurus/pytorch_text', 'size': '10325'}