<jupyter_start><jupyter_text>Data Explore[The SST-2 dataset](https://nlp.stanford.edu/sentiment/index.html) is a standard benchmark for [sentiment classification task](http://nlpprogress.com/english/sentiment_analysis.html). The SST-2 is also a part of the [GLUE Benchmark](https://gluebenchmark.com/tasks). To download the SST-2 dataset as part of GLUE data, you can run [this script](http://nlpprogress.com/english/sentiment_analysis.html). After running the script, the SST-2 could be found under ``glue_data/SST-2`` folder.<jupyter_code># to get the list of all pretrained BERT Language models run
nemo_nlp.nm.trainables.get_pretrained_lm_models_list()
WORK_DIR = 'output/'
DATA_DIR = 'glue_data/SST-2'

# To use mixed precision, set AMP_OPTIMIZATION_LEVEL to 'O1' or 'O2',
# to train without mixed precision, set it to 'O0'.
AMP_OPTIMIZATION_LEVEL = 'O1'
# select the model name from the list generated in the previous cell
PRETRAINED_MODEL_NAME = 'bert-base-uncased'
MAX_SEQ_LEN = 64 # we will pad with 0's shorter sentences and truncate longer
BATCH_SIZE = 256 # 64 for 'bert-large-uncased'
df = pd.read_csv(DATA_DIR + '/train.tsv', sep='\t')
test_df = pd.read_csv(DATA_DIR + '/test.tsv', sep='\t')
df.head()
test_df.head()<jupyter_output><empty_output><jupyter_text>The dataset comes with a train file (labeled) and a test file (not labeled).  We will use part of the train file for model validation<jupyter_code># Split train to train and val and save to disk
np.random.seed(123)
train_mask = np.random.rand((len(df))) < .8
train_df = df[train_mask]
val_df = df[~train_mask]<jupyter_output><empty_output><jupyter_text>In order to take advantage of NeMo's pre-built text classification data layer, the data should be formatted as "sentence\tlabel" (sentence tab label).<jupyter_code># We will add a label column with all 0's (but they will not be used for anything).
test_df['label'] = 0
test_df = test_df[['sentence', 'label']]
test_df.head()
# Save new train, val, and test to disk
SPLIT_DATA_DIR = os.path.join(DATA_DIR, 'split')

os.makedirs(SPLIT_DATA_DIR, exist_ok=True)

train_df.to_csv(os.path.join(SPLIT_DATA_DIR, 'train.tsv'), sep='\t', index=False)
val_df.to_csv(os.path.join(SPLIT_DATA_DIR, 'eval.tsv'), sep='\t', index=False)
test_df.to_csv(os.path.join(SPLIT_DATA_DIR, 'test.tsv'), sep='\t', index=False)<jupyter_output><empty_output><jupyter_text>Neural ModulesIn NeMo, everything is a Neural Module. Neural modules abstract data and neural network architectures. Where a deep learning framework like PyTorch or Tensorflow is used to combine neural network layers to create a neural network. NeMo is used to combine data and neural networks to create AI applications.The Neural Module Factory will then manage the neural modules, taking care to flow data through the neural modules, and is also responsible for training (including mixed precision and distributed), logging, and inference.<jupyter_code># instantiate the neural module factory
nf = nemo.core.NeuralModuleFactory(log_dir=WORK_DIR,
                                   create_tb_writer=True,
                                   add_time_to_log_dir=False,
                                   optimization_level=AMP_OPTIMIZATION_LEVEL)<jupyter_output><empty_output><jupyter_text>Pre-trained models will be automatically downloaded and cached.<jupyter_code># Pre-trained BERT
bert = model = nemo_nlp.nm.trainables.get_pretrained_lm_model(pretrained_model_name=PRETRAINED_MODEL_NAME)
tokenizer = nemo.collections.nlp.data.tokenizers.get_tokenizer(tokenizer_name='nemobert', pretrained_model_name=PRETRAINED_MODEL_NAME)<jupyter_output>[NeMo I 2020-05-28 17:10:07 bert_tokenizer:78] Deriving bert model type from pretrained model name.<jupyter_text>Note here that the BERT models we are working with are massive. This gives our models a large capacity for learning that is needed to understand the nuance and complexity of natural language.<jupyter_code>print(f'{PRETRAINED_MODEL_NAME} has {bert.num_weights} weights')<jupyter_output>bert-base-uncased has 109482240 weights<jupyter_text>Here we define and instantiate the feed forward network that takes as input our BERT embeddings. This network will be used to output the sentence classifications.<jupyter_code># mlp classifier
bert_hidden_size = bert.hidden_size

mlp = SequenceClassifier(hidden_size=bert_hidden_size, 
                         num_classes=2,
                         num_layers=2,
                         log_softmax=False,
                         dropout=0.1)

loss = CrossEntropyLossNM()
# Compared to the BERT model, the MLP is tiny.
print(f'MLP has {mlp.num_weights} weights')<jupyter_output>MLP has 592130 weights<jupyter_text>PipelinesPipelines are used to define how data will flow the different neural networks. In this case, our data will flow through the BERT network and then the MLP network.We also have different pipelines for training, validation, and inference data.  For training data, we want it to be used for optimization so it must be shuffled and we also need to compute the loss.For validation data, we won't use it for optimization but we want to know the loss.And for inference data, we only want the final predictions coming from the model.  Data Layers We can gain a lot of efficiency by saving the tokenized data to disk. For future model runs we then don't need to tokenize every time.<jupyter_code>USE_CACHE = True

train_data = BertTextClassificationDataLayer(input_file=os.path.join(SPLIT_DATA_DIR, 'train.tsv'),
                                             tokenizer=tokenizer,
                                             max_seq_length=MAX_SEQ_LEN,
                                             shuffle=True,
                                             batch_size=BATCH_SIZE,
                                             use_cache=USE_CACHE)

val_data = BertTextClassificationDataLayer(input_file=os.path.join(SPLIT_DATA_DIR, 'eval.tsv'),
                                           tokenizer=tokenizer,
                                           max_seq_length=MAX_SEQ_LEN,
                                           batch_size=BATCH_SIZE,
                                           use_cache=USE_CACHE)
train_input, train_token_types, train_attn_mask, train_labels = train_data()
val_input, val_token_types, val_attn_mask, val_labels = val_data()<jupyter_output><empty_output><jupyter_text>BERT Embeddings<jupyter_code>train_embeddings = bert(input_ids=train_input,
                        token_type_ids=train_token_types,
                        attention_mask=train_attn_mask)
val_embeddings = bert(input_ids=val_input,
                      token_type_ids=val_token_types,
                      attention_mask=val_attn_mask)<jupyter_output><empty_output><jupyter_text>Inspect BERT EmbeddingsIf we want to inspect the data as it flows through our neural factory we can use the .infer method.  This method will give us the tensors without performing any optimization.<jupyter_code>val_input_tensors = nf.infer(tensors=[val_input])
print(val_input_tensors[0][0][0])
%%time
val_embeddings_tensors = nf.infer(tensors=[val_embeddings])
# each word is embedded into bert_hidden_size space
# shape: BATCH_SIZE * MAX_SEQ_LEN * bert_hidden_size
val_embeddings_tensors[0][0].shape
print(val_embeddings_tensors[0][0][1][:, 0])<jupyter_output>tensor([-0.0419,  0.3524, -0.0894,  0.4333,  0.1847, -0.1055,  0.1183,  1.0781,
         1.6059,  0.7314, -0.0276,  0.3873, -0.1493,  0.2835,  0.1241,  0.3322,
        -0.0739,  0.9682,  0.6255,  1.8736,  0.2042, -0.8762,  1.2981,  0.1381,
         0.1281,  0.8677,  0.6106,  0.6792,  0.8891,  0.3673,  0.5610,  0.3335,
         0.6599,  0.5207,  0.2783,  0.2677,  0.2921,  0.5695,  0.6468,  0.6725,
         0.3356, -0.0210, -0.0821,  0.2344,  0.0618,  0.3286,  0.3372,  0.3234,
         0.0713, -0.1934,  0.4176, -0.0325,  0.1759,  0.5047,  0.0894,  0.5957,
         0.7829,  0.8567,  0.3592,  0.2224,  0.2628,  0.1901,  0.2108,  0.4541])<jupyter_text>Understanding and Visualizing BERT EmbeddingsWe are going to look at the BERT embeddings for the words (1-word sentences) in "SPLIT_DATA_DIR/positive_negative.tsv". Since the BERT embeddings are 768 dimensional for BERT base and 1024 dimensional for BERT large, we'll first apply TSNE and reduce the embeddings to two dimensions.<jupyter_code>spectrum_words = ['abysmal', 'apalling', 'dreadful', 'awful', 'terrible',
                  'very bad', 'really bad', 'rubbish', 'unsatisfactory',
                  'bad', 'poor', 'great', 'really good', 'very good', 'awesome'
                  'fantastic', 'superb', 'brilliant', 'incredible', 'excellent'
                  'outstanding', 'perfect']

spectrum_file = os.path.join(SPLIT_DATA_DIR, 'positive_negative.tsv')
with open(spectrum_file, 'w+') as f:
    f.write('sentence\tlabel')
    for word in spectrum_words:
        f.write('\n' + word + '\t0')
spectrum_df = pd.read_csv(spectrum_file, delimiter='\t')
print(spectrum_df.head())
# positive negative spectrum
spectrum_data = BertTextClassificationDataLayer(input_file=spectrum_file,
                                                tokenizer=tokenizer,
                                                max_seq_length=MAX_SEQ_LEN,
                                                batch_size=BATCH_SIZE)
spectrum_input, spectrum_token_types, spectrum_attn_mask, spectrum_labels = spectrum_data()
spectrum_embeddings = bert(input_ids=spectrum_input,
                           token_type_ids=spectrum_token_types,
                           attention_mask=spectrum_attn_mask)
spectrum_embeddings_tensors = nf.infer(tensors=[spectrum_embeddings])
spectrum_embeddings_tensors[0][0].shape
plt.figure(figsize=(100,100))
plt.imshow(spectrum_embeddings_tensors[0][0][:,0,:].numpy())
spectrum_activations = spectrum_embeddings_tensors[0][0][:,0,:].numpy()
tsne_spectrum = TSNE(n_components=2, perplexity=10, verbose=1, learning_rate=2,
                     random_state=123).fit_transform(spectrum_activations)

fig = plt.figure(figsize=(10,10))
plt.plot(tsne_spectrum[0:11, 0], tsne_spectrum[0:11, 1], 'rx')
plt.plot(tsne_spectrum[11:, 0], tsne_spectrum[11:, 1], 'bo')
for (x,y, label) in zip(tsne_spectrum[0:, 0], tsne_spectrum[0:, 1], spectrum_df.sentence.values.tolist() ):
    plt.annotate(label, # this is the text
                 (x,y), # this is the point to label
                 textcoords="offset points", # how to position the text
                 xytext=(0,10), # distance from text to points (x,y)
                 ha='center') # horizontal alignment can be left, right or center<jupyter_output>[t-SNE] Computing 19 nearest neighbors...
[t-SNE] Indexed 20 samples in 0.000s...
[t-SNE] Computed neighbors for 20 samples in 0.001s...
[t-SNE] Computed conditional probabilities for sample 20 / 20
[t-SNE] Mean sigma: 2.210090
[t-SNE] KL divergence after 100 iterations with early exaggeration: 35.170803
[t-SNE] KL divergence after 900 iterations: 0.119485<jupyter_text>Training Pipeline In order to optimize our network, we need to pass the embeddings through the MLP network and then compute the loss.<jupyter_code>train_logits = mlp(hidden_states=train_embeddings)
val_logits = mlp(hidden_states=val_embeddings)

train_loss = loss(logits=train_logits, labels=train_labels)
val_loss = loss(logits=val_logits, labels=val_labels)<jupyter_output><empty_output><jupyter_text>CallbacksCallbacks are used to record and log metrics and save checkpoints for the training and evaluation. We use callbacks to print to screen and also to tensorboard.<jupyter_code>NUM_EPOCHS = 1
NUM_GPUS = 1
LEARNING_RATE = 1e-5
OPTIMIZER = 'adam'

train_data_size = len(train_data)
steps_per_epoch = math.ceil(train_data_size / (BATCH_SIZE * NUM_GPUS))

train_callback = nemo.core.SimpleLossLoggerCallback(tensors=[train_loss, train_logits],
                            print_func=lambda x:nemo.logging.info(f'Train loss: {str(np.round(x[0].item(), 3))}'),
                            tb_writer=nf.tb_writer,
                            get_tb_values=lambda x: [["train_loss", x[0]]],
                            step_freq=steps_per_epoch)

eval_callback = nemo.core.EvaluatorCallback(eval_tensors=[val_logits, val_labels],
                                            user_iter_callback=lambda x, y: eval_iter_callback(x, y, val_data),
                                            user_epochs_done_callback=lambda x:
                                                eval_epochs_done_callback(x, f'{nf.work_dir}/graphs'),
                                            tb_writer=nf.tb_writer,
                                            eval_step=steps_per_epoch)

# Create callback to save checkpoints
ckpt_callback = nemo.core.CheckpointCallback(folder=nf.checkpoint_dir,
                                             epoch_freq=1)
lr_policy_fn = get_lr_policy('WarmupAnnealing',
                             total_steps=NUM_EPOCHS * steps_per_epoch,
                             warmup_ratio=0.1)
%%time
nf.train(tensors_to_optimize=[train_loss],
         callbacks=[train_callback, eval_callback, ckpt_callback],
         lr_policy=lr_policy_fn,
         optimizer=OPTIMIZER,
         optimization_params={'num_epochs': NUM_EPOCHS, 'lr': LEARNING_RATE})<jupyter_output>Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
[NeMo I 2020-05-28 17:10:19 deprecated_callbacks:195] Starting .....
[NeMo I 2020-05-28 17:10:20 callbacks:495] Found 2 modules with weights:
[NeMo I 2020-05-28 17:10:20 callbacks:497] SequenceClassifier
[NeMo I 2020-05-28 17:10:20 callbacks:497] BERT
[NeMo I 2020-05-28 17:10:20 callbacks:498] Total model parame[...]<jupyter_text>Multi-Gpu TrainingRESTART KERNEL BEFORE RUNNING THE MULTI-GPU TRAINING<jupyter_code>%%time
num_gpus = 4
!python -m torch.distributed.launch --nproc_per_node=$NUM_GPUS text_classification_with_bert.py \
--pretrained_model_name $PRETRAINED_MODEL_NAME \
--data_dir $SPLIT_DATA_DIR \
--train_file_prefix 'train' \
--eval_file_prefix 'eval' \
--use_cache \
--batch_size $BATCH_SIZE \
--max_seq_length 64 \
--num_gpus $NUM_GPUS \
--num_epochs $NUM_EPOCHS \
--amp_opt_level $AMP_OPTIMIZATION_LEVEL \
--work_dir $WORK_DIR<jupyter_output>/home/ebakhturina/anaconda3/envs/nemo/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ebakhturina/anaconda3/envs/nemo/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ebakhturina/anaconda3/envs/nemo/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ebakhturina/anaconda3/envs/nemo[...]<jupyter_text>Inference PipelineFor inference we instantiate the same neural modules but now we will be using the checkpoints that we just learned.<jupyter_code>test_data = BertTextClassificationDataLayer(input_file=os.path.join(SPLIT_DATA_DIR, 'test.tsv'),
                                            tokenizer=tokenizer,
                                            max_seq_length=MAX_SEQ_LEN,
                                            batch_size=BATCH_SIZE)
test_input, test_token_types, test_attn_mask, _ = test_data()
test_embeddings = bert(input_ids=test_input,
                        token_type_ids=test_token_types,
                        attention_mask=test_attn_mask)
test_logits = mlp(hidden_states=test_embeddings)
%%time
test_logits_tensors = nf.infer(tensors=[test_logits])
test_probs = torch.nn.functional.softmax(torch.cat(test_logits_tensors[0])).numpy()[:, 1] 
test_df = pd.read_csv(os.path.join(SPLIT_DATA_DIR, 'test.tsv'), sep='\t')
test_df['prob'] = test_probs 
inference_file = os.path.join(SPLIT_DATA_DIR, 'test_inference.tsv')
test_df.to_csv(inference_file, sep='\t', index=False)
def sample_classification(data_path):
    df = pd.read_csv(data_path, sep='\t')
    sample = df.sample()
    sentence = sample.sentence.values[0]
    prob = sample.prob.values[0]
    result = f'{sentence} | {prob}'
    return result
num_samples = 10
for _ in range(num_samples):
    print(sample_classification(inference_file))<jupyter_output>the story ... is moldy and obvious . | 0.026556572
this idea has lost its originality ... and neither star appears very excited at rehashing what was basically a one-joke picture . | 0.031499382
the plot grinds on with yawn-provoking dullness . | 0.030588808
an offbeat , sometimes gross and surprisingly appealing animated film about the true meaning of the holidays . | 0.9845458
at first , the sight of a blind man directing a film is hilarious , but as the film goes on , the joke wears thin . | 0.07857165
like the film 's almost anthropologically detailed realization of early - '80s suburbia , it 's significant without being overstated . | 0.98240626
there 's something fundamental missing from this story : something or someone to care about . | 0.17831111
thoughtful , even stinging at times , and lots of fun . | 0.9891187
demands too much of most viewers . | 0.036356665
a romantic comedy , yes , but one with characters who think and talk about their goals , and are working on hard deci[...]<jupyter_text>Inference Results:the film is just a big , gorgeous , mind-blowing , breath-taking mess . | 0.2738656a sensual performance from abbass buoys the flimsy story , but her inner journey is largely unexplored and we 're left wondering about this exotic-looking woman whose emotional depths are only hinted at . | 0.48260054  Single sentence classification<jupyter_code>def classify_sentence(nf, tokenizer, bert, mlp, sentence):
    sentence = sentence.lower()
    tmp_file = "/tmp/tmp_sentence.tsv"
    with open(tmp_file, 'w+') as tmp_tsv:
        header = 'sentence\tlabel\n'
        line = sentence + '\t0\n'
        tmp_tsv.writelines([header, line])

    tmp_data = BertTextClassificationDataLayer(input_file=tmp_file,
                                               tokenizer=tokenizer,
                                               max_seq_length=128,
                                               batch_size=1)
    
    tmp_input, tmp_token_types, tmp_attn_mask, _ = tmp_data()
    tmp_embeddings = bert(input_ids=tmp_input,
                          token_type_ids=tmp_token_types,
                          attention_mask=tmp_attn_mask)
    tmp_logits = mlp(hidden_states=tmp_embeddings)
    tmp_logits_tensors = nf.infer(tensors=[tmp_logits, tmp_embeddings])
    tmp_probs = torch.nn.functional.softmax(torch.cat(tmp_logits_tensors[0])).numpy()[:, 1] 
    print(f'{sentence} | {tmp_probs[0]}')
sentences = ['point break is the best movie of all time',
             'the movie was a wonderful exercise in understanding the struggles of native americans',
             'the performance of diego luna had me excited and annoyed at the same time',
             '<NAME> is the only good thing about this film']

for sentence in sentences:
    classify_sentence(nf, tokenizer, bert, mlp, sentence)<jupyter_output>[NeMo I 2020-05-28 17:13:26 text_classification_dataset:82] /tmp/tmp_sentence.tsv: 1
[NeMo I 2020-05-28 17:13:26 data_preprocessing:250] Min: 11 |                  Max: 11 |                  Mean: 11.0 |                  Median: 11.0
[NeMo I 2020-05-28 17:13:26 data_preprocessing:252] 75 percentile: 11.0
[NeMo I 2020-05-28 17:13:26 data_preprocessing:253] 99 percentile: 11.0
[NeMo I 2020-05-28 17:13:26 text_classification_dataset:118] 0 out of 1                         sentences with more than 128 subtokens.
[NeMo I 2020-05-28 17:13:26 text_classification_dataset:175] *** Example ***
[NeMo I 2020-05-28 17:13:26 text_classification_dataset:176] example_index: 0
[NeMo I 2020-05-28 17:13:26 text_classification_dataset:177] subtokens: [CLS] point break is the best movie of all time [SEP]
[NeMo I 2020-05-28 17:13:26 text_classification_dataset:178] sent_label: 0
[NeMo I 2020-05-28 17:13:26 text_classification_dataset:179] input_ids: 101 2391 3338 2003 1996 2190 3185 1997 2035 2051 102 0 0 0[...]<jupyter_text>Understanding and Visualizing BERT EmbeddingsNow that we've fine-tuned our BERT model, let's see if the word embeddings have changed.<jupyter_code>spectrum_embeddings = bert(input_ids=spectrum_input,
                           token_type_ids=spectrum_token_types,
                           attention_mask=spectrum_attn_mask)

spectrum_embeddings_tensors = nf.infer(tensors=[spectrum_embeddings])

plt.figure(figsize=(100,100))
plt.imshow(spectrum_embeddings_tensors[0][0][:,0,:].numpy())
spectrum_activations = spectrum_embeddings_tensors[0][0][:,0,:].numpy()
tsne_spectrum = TSNE(n_components=2, perplexity=10, verbose=1, learning_rate=2,
                     random_state=123).fit_transform(spectrum_activations)

fig = plt.figure(figsize=(10,10))
plt.plot(tsne_spectrum[0:11, 0], tsne_spectrum[0:11, 1], 'rx')
plt.plot(tsne_spectrum[11:, 0], tsne_spectrum[11:, 1], 'bo')
for (x,y, label) in zip(tsne_spectrum[0:, 0], tsne_spectrum[0:, 1], spectrum_df.sentence.values.tolist() ):
    plt.annotate(label, # this is the text
                 (x,y), # this is the point to label
                 textcoords="offset points", # how to position the text
                 xytext=(0,10), # distance from text to points (x,y)
                 ha='center') # horizontal alignment can be left, right or center<jupyter_output>[t-SNE] Computing 19 nearest neighbors...
[t-SNE] Indexed 20 samples in 0.000s...
[t-SNE] Computed neighbors for 20 samples in 0.001s...
[t-SNE] Computed conditional probabilities for sample 20 / 20
[t-SNE] Mean sigma: 11.588557
[t-SNE] KL divergence after 250 iterations with early exaggeration: 33.095665
[t-SNE] KL divergence after 800 iterations: 0.001649
{'fraction_non_alphanumeric': 0.0979254571026723, 'fraction_numerical': 0.047995780590717296, 'mean_word_length': 3.367178502879079, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 55, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '358455', 'license': 'Apache-2.0', 'n_tokens_mistral': 8052, 'n_tokens_neox': 7201, 'n_words': 1900, 'path': 'examples/nlp/text_classification/sentiment_analysis_with_bert.ipynb', 'repo_name': 'shangw-nvidia/NeMo', 'size': '22756'}