<jupyter_start><jupyter_text>IntroductionBioBERT has the same network architecture as the original BERT, but instead of Wikipedia and BookCorpus it is pretrained on PubMed, a large biomedical text corpus, which achieves better performance in biomedical downstream tasks, such as question answering(QA), named entity recognition(NER) and relationship extraction(RE). This model was trained for 1M steps. For more information please refer to the original paper https://academic.oup.com/bioinformatics/article/36/4/1234/5566506.  For details about BERT please refer to https://ngc.nvidia.com/catalog/models/nvidia:bertbaseuncasedfornemo.In this notebook we're going to showcase how to train BioBERT on a biomedical relation extraction (RE) dataset.  Download model checkpointDownload BioBert/BioMegatron checkpoints from  NGC: https://ngc.nvidia.com/catalog/models and put the encoder weights at `./checkpoints/biobert/BERT.pt` or `./checkpoints/biomegatron/BERT.pt` and the model configuration file at `./checkpoints/biobert/bert_config.json` or `./checkpoints/biomegatron/bert_config.json`. For BioBERT download e.g. https://ngc.nvidia.com/catalog/models/nvidia:biobertbasecasedfornemo.For BioMegatron download e.g. https://ngc.nvidia.com/catalog/models/nvidia:biomegatron345muncased.<jupyter_code># Set which model to use.
model_type="biobert" # "biomegatron"
base_checkpoint_path={'biobert': './checkpoints/biobert/', 'biomegatron': './checkpoints/biomegatron/'}
pretrained_model_name={'biobert': 'bert-base-cased', 'biomegatron': 'megatron-bert-uncased'}
do_lower_case={'biobert': False, 'biomegatron': True}
work_dir={'biobert': 'output_re_biobert', 'biomegatron': 'output_re_biomegatron'}<jupyter_output><empty_output><jupyter_text><jupyter_code># the checkpoints are available from NGC: https://ngc.nvidia.com/catalog/models
CHECKPOINT_ENCODER = os.path.join(base_checkpoint_path[model_type], 'BERT.pt') # Model encoder checkpoint file
CHECKPOINT_CONFIG = os.path.join(base_checkpoint_path[model_type], 'bert_config.json') # Model configuration file
    
if not os.path.exists(CHECKPOINT_ENCODER):
    raise OSError(errno.ENOENT, os.strerror(errno.ENOENT), CHECKPOINT_ENCODER)

if not os.path.exists(CHECKPOINT_CONFIG):
    raise OSError(errno.ENOENT, os.strerror(errno.ENOENT), CHECKPOINT_CONFIG)<jupyter_output><empty_output><jupyter_text>Download training dataIn this example we download the RE dataset chemprot to ./datasets/chemprot and process it with text_classification/data/import_datasets.py<jupyter_code>#download https://github.com/arwhirang/recursive_chemprot/blob/master/Demo/tree_LSTM/data/chemprot-data_treeLSTM.zip and extract it into ./datasets/chemprot
data_dir="./datasets"
dataset="chemprot"
if not os.path.exists(f"{data_dir}/{dataset}"):
    !mkdir -p $data_dir/$dataset
    !wget "https://github.com/arwhirang/recursive_chemprot/blob/master/Demo/tree_LSTM/data/chemprot-data_treeLSTM.zip?raw=true" -O data.zip
    !unzip data.zip -d $data_dir/$dataset
    !rm data.zip

!python ../text_classification/data/import_datasets.py --source_data_dir=$data_dir/$dataset --target_data_dir=$data_dir/$dataset --dataset_name=$dataset
!ls -l $data_dir/$dataset<jupyter_output><empty_output><jupyter_text>After the previous step, you should have a ./datasets/chemprot folder that contains the following files:- train.tsv- test.tsv- dev.tsv- label_mapping.tsvThe format of the data described in NeMo docs.  Create Neural Modules<jupyter_code>model_checkpoint=CHECKPOINT_ENCODER # language model encoder file
model_config=CHECKPOINT_CONFIG # model configuration file
work_dir=work_dir[model_type]
train_data_text_file=f"{data_dir}/{dataset}/train.tsv"
eval_data_text_file=f"{data_dir}/{dataset}/dev.tsv"
fc_dropout=0.1
max_seq_length=128
batch_size=32
num_output_layers=1

nf = nemo.core.NeuralModuleFactory(
    placement=nemo.core.DeviceType.GPU
)
model = nemo_nlp.nm.trainables.get_pretrained_lm_model(
        config=model_config, pretrained_model_name=pretrained_model_name[model_type], checkpoint=model_checkpoint
    )
tokenizer = nemo.collections.nlp.data.tokenizers.get_tokenizer(
    tokenizer_name='nemobert',
    pretrained_model_name=pretrained_model_name[model_type],
    do_lower_case=do_lower_case[model_type]
)
hidden_size = model.hidden_size
data_desc = TextClassificationDataDesc(data_dir=f"{data_dir}/{dataset}", modes=['train', 'dev'])
classifier = nemo_nlp.nm.trainables.SequenceClassifier(    
    hidden_size=hidden_size,
    num_classes=data_desc.num_labels,
    dropout=fc_dropout,
    num_layers=num_output_layers,
    log_softmax=False,
)
task_loss = CrossEntropyLossNM(weight=None)
train_data_layer = BertTextClassificationDataLayer(
    tokenizer=tokenizer,
    input_file=train_data_text_file,
    max_seq_length=max_seq_length,
    batch_size=batch_size,
    shuffle=True,
    use_cache=True
)
eval_data_layer = BertTextClassificationDataLayer(
    tokenizer=tokenizer,
    input_file=eval_data_text_file,
    max_seq_length=max_seq_length,
    batch_size=batch_size,
    shuffle=False,
    use_cache=False
)<jupyter_output><empty_output><jupyter_text>Creating Neural graph<jupyter_code>train_data = train_data_layer()
train_hidden_states = model(input_ids=train_data.input_ids, token_type_ids=train_data.input_type_ids, attention_mask=train_data.input_mask)
train_logits = classifier(hidden_states=train_hidden_states)
loss = task_loss(logits=train_logits, labels=train_data.labels)
# If you're training on multiple GPUs, this should be
# len(train_data_layer) // (batch_size * batches_per_step * num_gpus)
train_steps_per_epoch = len(train_data_layer) // batch_size
logging.info(f"doing {train_steps_per_epoch} steps per epoch")

eval_data = eval_data_layer()
eval_hidden_states = model(input_ids=eval_data.input_ids, token_type_ids=eval_data.input_type_ids, attention_mask=eval_data.input_mask)
eval_logits = classifier(hidden_states=eval_hidden_states)<jupyter_output><empty_output><jupyter_text>Create Callbacks<jupyter_code>train_callback = nemo.core.SimpleLossLoggerCallback(
    tensors=[loss],
    print_func=lambda x: logging.info("Loss: {:.3f}".format(x[0].item())),
    get_tb_values=lambda x: [["loss", x[0]]],
    step_freq=100,
    tb_writer=nf.tb_writer,
)

# Callback to evaluate the model
eval_callback = nemo.core.EvaluatorCallback(
        eval_tensors=[eval_logits, eval_data.labels],
        user_iter_callback=lambda x, y: eval_iter_callback(x, y, eval_data_layer),
        user_epochs_done_callback=lambda x: eval_epochs_done_callback(x, f'{nf.work_dir}/graphs'),
        tb_writer=nf.tb_writer,
        eval_step=500,
    )<jupyter_output><empty_output><jupyter_text>TrainingTraining could take several minutes.<jupyter_code>num_epochs=3
lr_warmup_proportion=0.1
lr=3e-5
weight_decay=0.01
lr_policy_fn = get_lr_policy("WarmupAnnealing", total_steps=num_epochs * train_steps_per_epoch, warmup_ratio=lr_warmup_proportion
)
nf.train(
    tensors_to_optimize=[loss],
    callbacks=[train_callback, eval_callback],
    lr_policy=lr_policy_fn,
    optimizer="adam_w",
    optimization_params={"num_epochs": num_epochs, "lr": lr, "weight_decay": weight_decay},
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10592509782001118, 'fraction_numerical': 0.006008943543879262, 'mean_word_length': 6.796296296296297, 'pattern_counts': {'":': 3, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '256340', 'license': 'Apache-2.0', 'n_tokens_mistral': 2529, 'n_tokens_neox': 2446, 'n_words': 417, 'path': 'examples/nlp/biobert_notebooks/biobert_re.ipynb', 'repo_name': 'HerringForks/NeMo', 'size': '7156'}