<jupyter_start><jupyter_text>Sentiment Analysis (SA) with pretrained Language Model (LM) In this notebook, we are going to build a sentiment analysis model based on the pretrained language model. We are focusing on the best usability to support traditional nlp tasks in a simple fashion. The building process is simple three steps. Let us get started now.We use movie reviews from the Large Movie Review Dataset, as known as the IMDB dataset. In this task, given a moview, the model attemps to predict its sentiment, which can be positive or negative.  Preparation and settings  Load mxnet and gluonnlp<jupyter_code>import random
import time
import multiprocessing as mp
import numpy as np

import mxnet as mx
from mxnet import nd, gluon, autograd

import gluonnlp as nlp

random.seed(123)
np.random.seed(123)
mx.random.seed(123)<jupyter_output><empty_output><jupyter_text>Hyperparameters Our model is based on a standard LSTM model. We use a hidden size of 200. We use bucketing for speeding up the processing of variable-length sequences. To enable multi-gpu training, we can simply change num_gpus to some value larger than 1.<jupyter_code>dropout = 0
language_model_name = 'standard_lstm_lm_200'
pretrained = True
num_gpus = 1
learning_rate = 0.005 * num_gpus
batch_size = 16 * num_gpus
bucket_num = 10
bucket_ratio = 0.2
epochs = 1
grad_clip = None
log_interval = 100
context = [mx.gpu(i) for i in range(num_gpus)] if num_gpus else [mx.cpu()]<jupyter_output><empty_output><jupyter_text>Sentiment analysis model with pre-trained language model encoder The model architecture is based on pretrained LM:![sa-model](samodel-v3.png) Our model is composed of a two-layer LSTM followed by an average pooling and a sigmoid output layer as illustrated in the Figure above. From the embedding layer, the new representations will be passed to LSTM cells. These will include information about the sequence of words in the data. Thus, given an input sequence, the memory cells in the LSTM layer will produce a representation sequence. This representation sequence is then averaged over all timesteps resulting in representation h. Finally, this representation is fed to a sigmoid output layer. We’re using the sigmoid  because we’re trying to predict if this text has positive or negative sentiment, and a sigmoid activation function allows the model to compute the posterior probability.<jupyter_code>class SentimentNet(gluon.Block):
    def __init__(self, embedding_block, encoder_block, dropout, 
                 prefix=None, params=None):
        super(SentimentNet, self).__init__(prefix=prefix, params=params)
        with self.name_scope():
            self.embedding = embedding_block
            self.encoder = encoder_block
            self.out_layer = gluon.nn.HybridSequential()
            with self.out_layer.name_scope():
                self.out_layer.add(gluon.nn.Dropout(dropout))
                self.out_layer.add(gluon.nn.Dense(1, flatten=False))

    def forward(self, data, valid_length):
        encoded = self.encoder(nd.Dropout(self.embedding(data), 
                                          0.2, axes=(0,)))  # Shape(T, N, C)
        # Zero out the values with position exceeding the valid length. 
        masked_encoded = nd.SequenceMask(encoded,
                                         sequence_length=valid_length,
                                         use_sequence_length=True)
        agg_state = nd.broadcast_div(nd.sum(masked_encoded, axis=0),
                                     nd.expand_dims(valid_length, axis=1))
        out = self.out_layer(agg_state)
        return out


lm_model, vocab = nlp.model.get_model(name=language_model_name,
                                      dataset_name='wikitext-2',
                                      pretrained=pretrained,
                                      ctx=context,
                                      dropout=dropout)
net = SentimentNet(embedding_block=lm_model.embedding, 
                   encoder_block=lm_model.encoder,
                   dropout=dropout)
net.out_layer.initialize(mx.init.Xavier(), ctx=context)
net.hybridize()
print(net)<jupyter_output>SentimentNet(
  (encoder): LSTM(200 -> 200.0, TNC, num_layers=2)
  (embedding): HybridSequential(
    (0): Embedding(33278 -> 200, float32)
  )
  (out_layer): HybridSequential(
    (0): Dropout(p = 0, axes=())
    (1): Dense(None -> 1, linear)
  )
)<jupyter_text>In the above code, we first acquire a pretrained model on Wikitext-2 dataset using nlp.model.get_model. We then construct a SentimentNet object, which takes as input the embedding layer and encoder of the pretrained model. As we employ the pretrained embedding layer and encoder, we only need to initialize the output layer using net.out_layer.initialize(mx.init.Xavier(), ctx=context).  Data pipeline  Load sentiment analysis dataset -- IMDB reviews<jupyter_code># train_dataset and test_dataset are both SimpleDataset objects, 
# which is a wrapper for lists and arrays.
train_dataset, test_dataset = [nlp.data.IMDB(segment=segment) 
                               for segment in ('train', 'test')]
print("Tokenize using spaCy...")
# tokenizer takes as input a string and outputs a list of tokens.
tokenizer = nlp.data.SpacyTokenizer('en')
# length_clip takes as input a list and outputs a list with maximum length 500.
length_clip = nlp.data.ClipSequence(500)

def preprocess(x):
    data, label = x
    # In the labeled train/test sets, a negative review has a score <= 4 
    # out of 10, and a positive review has a score >= 7 out of 10. Thus 
    # reviews with more neutral ratings are not included in the train/test 
    # sets. We labeled a negative review whose score <= 4 as 0, and a 
    # positive review whose score >= 7 as 1. As the neural ratings are not 
    # included in the datasets, we can simply use 5 as our threshold. 
    label = int(label > 5)
    # A token index or a list of token indices is 
    # returned according to the vocabulary.
    data = vocab[length_clip(tokenizer(data))]
    return data, label, float(len(data))

def get_length(x):
    return x[2]

def preprocess_dataset(dataset):
    start = time.time()
    with mp.Pool() as pool:
        # Each sample is processed in an asynchronous manner.
        dataset = gluon.data.SimpleDataset(pool.map(preprocess, dataset))
        lengths = gluon.data.SimpleDataset(pool.map(get_length, dataset))
    end = time.time()
    print('Done! Tokenizing Time={:.2f}s, #Sentences={}'
          .format(end - start, len(dataset)))
    return dataset, lengths

train_dataset, train_data_lengths = preprocess_dataset(train_dataset)
test_dataset, test_data_lengths = preprocess_dataset(test_dataset)<jupyter_output>Tokenize using spaCy...
Done! Tokenizing Time=13.36s, #Sentences=25000
Done! Tokenizing Time=12.95s, #Sentences=25000<jupyter_text>Training  Evaluation using loss and accuracy<jupyter_code>def evaluate(net, dataloader, context):
    loss = gluon.loss.SigmoidBCELoss()
    total_L = 0.0
    total_sample_num = 0
    total_correct_num = 0
    start_log_interval_time = time.time()
    print('Begin Testing...')
    for i, (data, label, valid_length) in enumerate(dataloader):
        data = mx.nd.transpose(data.as_in_context(context))
        valid_length = valid_length.as_in_context(context).astype(np.float32)
        label = label.as_in_context(context)
        output = net(data, valid_length)
        L = loss(output, label)
        pred = (output > 0.5).reshape(-1)
        total_L += L.sum().asscalar()
        total_sample_num += label.shape[0]
        total_correct_num += (pred == label).sum().asscalar()
        if (i + 1) % log_interval == 0:
            print('[Batch {}/{}] elapsed {:.2f} s'.format(
                i + 1, len(dataloader), 
                time.time() - start_log_interval_time))
            start_log_interval_time = time.time()
    avg_L = total_L / float(total_sample_num)
    acc = total_correct_num / float(total_sample_num)
    return avg_L, acc<jupyter_output><empty_output><jupyter_text>In the following code, we use FixedBucketSampler, which assigns each data sample to a fixed bucket based on its length. The bucket keys are either given or generated from the input sequence lengths and number of the buckets.<jupyter_code>def train(net, context, epochs):
    trainer = gluon.Trainer(net.collect_params(), 
                            'ftml', 
                            {'learning_rate': learning_rate})
    loss = gluon.loss.SigmoidBCELoss()

    # Construct the DataLoader
    # Pad data, stack label and lengths
    batchify_fn = nlp.data.batchify.Tuple(nlp.data.batchify.Pad(axis=0), 
                                          nlp.data.batchify.Stack(),
                                          nlp.data.batchify.Stack())
    batch_sampler = nlp.data.sampler.FixedBucketSampler(train_data_lengths,
                                                        batch_size=batch_size,
                                                        num_buckets=bucket_num,
                                                        ratio=bucket_ratio,
                                                        shuffle=True)
    print(batch_sampler.stats())
    train_dataloader = gluon.data.DataLoader(dataset=train_dataset,
                                             batch_sampler=batch_sampler,
                                             batchify_fn=batchify_fn)
    test_dataloader = gluon.data.DataLoader(dataset=test_dataset,
                                            batch_size=batch_size,
                                            shuffle=False,
                                            batchify_fn=batchify_fn)
    parameters = net.collect_params().values()

    # Training/Testing
    for epoch in range(epochs):
        # Epoch training stats
        start_epoch_time = time.time()
        epoch_L = 0.0
        epoch_sent_num = 0
        epoch_wc = 0
        # Log interval training stats
        start_log_interval_time = time.time()
        log_interval_wc = 0
        log_interval_sent_num = 0
        log_interval_L = 0.0

        for i, (data, label, length) in enumerate(train_dataloader):
            if data.shape[0] > len(context):
                # Multi-gpu training. 
                data_list, label_list, length_list \
                = [gluon.utils.split_and_load(x, 
                                              context, 
                                              batch_axis=0, 
                                              even_split=False) 
                   for x in [data, label, length]]
            else:
                data_list = [data.as_in_context(context[0])]
                label_list = [label.as_in_context(context[0])]
                length_list = [length.as_in_context(context[0])]
            L = 0
            wc = length.sum().asscalar()
            log_interval_wc += wc
            epoch_wc += wc
            log_interval_sent_num += data.shape[1]
            epoch_sent_num += data.shape[1]
            for data, label, valid_length in zip(data_list, label_list, length_list):
                valid_length = valid_length
                with autograd.record():
                    output = net(data.T, valid_length)
                    L = L + loss(output, label).mean().as_in_context(context[0])
            L.backward()
            # Clip gradient
            if grad_clip:
                gluon.utils.clip_global_norm([p.grad(x.context) 
                                              for p in parameters for x in data_list],
                                             grad_clip)
            # Update parameter
            trainer.step(1)
            log_interval_L += L.asscalar()
            epoch_L += L.asscalar()
            if (i + 1) % log_interval == 0:
                print('[Epoch {} Batch {}/{}] elapsed {:.2f} s, \
                      avg loss {:.6f}, throughput {:.2f}K wps'.format(
                    epoch, i + 1, len(train_dataloader), 
                    time.time() - start_log_interval_time,
                    log_interval_L / log_interval_sent_num,
                    log_interval_wc / 1000 / (time.time() - start_log_interval_time)))
                # Clear log interval training stats
                start_log_interval_time = time.time()
                log_interval_wc = 0
                log_interval_sent_num = 0
                log_interval_L = 0
        end_epoch_time = time.time()
        test_avg_L, test_acc = evaluate(net, test_dataloader, context[0])
        print('[Epoch {}] train avg loss {:.6f}, test acc {:.2f}, \
        test avg loss {:.6f}, throughput {:.2f}K wps'.format(
            epoch, epoch_L / epoch_sent_num,
            test_acc, test_avg_L, epoch_wc / 1000 / 
            (end_epoch_time - start_epoch_time)))
train(net, context, epochs)
net(mx.nd.reshape(mx.nd.array(vocab[['This', 'movie', 'is', 'amazing']], 
                              ctx=context[0]), shape=(-1, 1)),
    mx.nd.array([4], ctx=context[0])).sigmoid()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08680288830849593, 'fraction_numerical': 0.012444307881394992, 'mean_word_length': 2.2458239840438794, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 44, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '117069', 'license': 'Apache-2.0', 'n_tokens_mistral': 3831, 'n_tokens_neox': 3606, 'n_words': 1052, 'path': 'docs/examples/sentiment_analysis/sentiment_analysis.ipynb', 'repo_name': 'zhiheng-huang/gluon-nlp', 'size': '13018'}