<jupyter_start><jupyter_text>In this tutorial, we are going to describe how to finetune BioMegatron - a [BERT](https://arxiv.org/abs/1810.04805)-like [Megatron-LM](https://arxiv.org/pdf/1909.08053.pdf) model pre-trained on large biomedical text corpus ([PubMed](https://pubmed.ncbi.nlm.nih.gov/) abstracts and full-text commercial use collection) - on the [NCBI Disease Dataset](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3951655/) for Named Entity Recognition.The model size of Megatron-LM can be larger than BERT, up to multi-billion parameters, compared to 345 million parameters of BERT-large.There are some alternatives of BioMegatron, most notably [BioBERT](https://arxiv.org/abs/1901.08746). Compared to BioBERT BioMegatron is larger by model size and pre-trained on larger text corpus.A more general tutorial of using BERT-based models, including Megatron-LM, for downstream natural language processing tasks can be found [here](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/01_Pretrained_Language_Models_for_Downstream_Tasks.ipynb).  Task Description**Named entity recognition (NER)**, also referred to as entity chunking, identification or extraction, is the task of detecting and classifying key information (entities) in text.For instance, **given sentences from medical abstracts, what diseases are mentioned?**In this case, our data input is sentences from the abstracts, and our labels are the precise locations of the named disease entities.  Take a look at the information provided for the dataset.For more details and general examples on Named Entity Recognition, please refer to the [Token Classification and Named Entity Recognition tutorial notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Token_Classification_Named_Entity_Recognition.ipynb).  DatasetThe [NCBI-disease corpus](https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/) is a set of 793 PubMed abstracts, annotated by 14 annotators. The annotations take the form of HTML-style tags inserted into the abstract text using the clearly defined rules.  The annotations identify named diseases, and can be used to fine-tune a language model to identify disease mentions in future abstracts, *whether those diseases were part of the original training set or not*. Here's an example of what an annotated abstract from the corpus looks like:```html10021369	Identification of APC2, a homologue of the adenomatous polyposis coli tumour suppressor .	The adenomatous polyposis coli ( APC ) tumour-suppressor protein controls the Wnt signalling pathway by forming a complex with glycogen synthase kinase 3beta ( GSK-3beta ) , axin / conductin and betacatenin . Complex formation induces the rapid degradation of betacatenin . In colon carcinoma cells , loss of APC leads to the accumulation of betacatenin in the nucleus , where it binds to and activates the Tcf-4 transcription factor ( reviewed in [ 1 ] [ 2 ] ) . Here , we report the identification and genomic structure of APC homologues . Mammalian APC2 , which closely resembles APC in overall domain structure , was functionally analyzed and shown to contain two SAMP domains , both of which are required for binding to conductin . Like APC , APC2 regulates the formation of active betacatenin-Tcf complexes , as demonstrated using transient transcriptional activation assays in APC - / - colon carcinoma cells . Human APC2 maps to chromosome 19p13 . 3 . APC and APC2 may therefore have comparable functions in development and cancer .``` In this example, we see the following tags within the abstract:```htmladenomatous polyposis coli tumouradenomatous polyposis coli ( APC ) tumourcolon carcinomacolon carcinomacancer``` For our purposes, we will consider any identified category (such as "Modifier", "Specific Disease", and a few others) to generally be a "disease".Let's download the dataset.<jupyter_code>DATA_DIR = "DATA_DIR"
os.makedirs(DATA_DIR, exist_ok=True)
os.makedirs(os.path.join(DATA_DIR, 'NER'), exist_ok=True)
print('Downloading NCBI data...')
wget.download('https://www.ncbi.nlm.nih.gov/CBBresearch/Dogan/DISEASE/NCBI_corpus.zip', DATA_DIR)
! unzip -o {DATA_DIR}/NCBI_corpus.zip -d {DATA_DIR}<jupyter_output><empty_output><jupyter_text>If you want to see more examples, you can explore the text of the corpus using the file browser to the left, or open files directly, for example typing a command like the following in a code-cell:! head -1 $DATA_DIR/NCBI_corpus_testing.txt We have two datasets derived from this corpus:  a text classification dataset and a named entity recognition (NER) dataset.  The text classification dataset labels the abstracts among three broad disease groupings.  We'll use this simple split to demonstrate the NLP text classification task.   The NER dataset labels individual words as diseases.  This dataset will be used for the NLP NER task.    Pre-process datasetA pre-processed NCBI-disease dataset for NER can be found [here](https://github.com/spyysalo/ncbi-disease/tree/master/conll) or [here](https://github.com/dmis-lab/biobertdatasets).We download the files under {DATA_DIR/NER} directory.<jupyter_code>wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/train.tsv', os.path.join(DATA_DIR, 'NER'))
wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/devel.tsv', os.path.join(DATA_DIR, 'NER'))
wget.download('https://raw.githubusercontent.com/spyysalo/ncbi-disease/master/conll/test.tsv', os.path.join(DATA_DIR, 'NER'))
NER_DATA_DIR = 'DATA_DIR/NER'
!ls -lh $NER_DATA_DIR<jupyter_output><empty_output><jupyter_text>Convert these to a format that is compatible with [NeMo Token Classification module](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/token_classification_train.py), using the [conversion script](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/import_from_iob_format.py).<jupyter_code>! mv DATA_DIR/NER/devel.tsv DATA_DIR/NER/dev.tsv
wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/data/import_from_iob_format.py')
! python import_from_iob_format.py --data_file=DATA_DIR/NER/train.tsv
! python import_from_iob_format.py --data_file=DATA_DIR/NER/dev.tsv
! python import_from_iob_format.py --data_file=DATA_DIR/NER/test.tsv<jupyter_output><empty_output><jupyter_text>The NER task requires two files: the text sentences, and the labels.  Run the next two cells to see a sample of the two files.<jupyter_code>!head $NER_DATA_DIR/text_train.txt
!head $NER_DATA_DIR/labels_train.txt<jupyter_output><empty_output><jupyter_text>IOB TaggingWe can see that the abstract has been broken into sentences.  Each sentence is then further parsed into words with labels that correspond to the original HTML-style tags in the corpus. The sentences and labels in the NER dataset map to each other with _inside, outside, beginning (IOB)_ tagging. Anything separated by white space is a word, including punctuation.  For the first sentence we have the following mapping:```textIdentification of APC2 , a homologue of the adenomatous polyposis coli tumour suppressor .O              O  O    O O O         O  O   B           I         I    I      O          O  ```Recall the original corpus tags:```htmlIdentification of APC2, a homologue of the adenomatous polyposis coli tumour suppressor .```The beginning word of the tagged text, "adenomatous", is now IOB-tagged with a B (beginning) tag, the other parts of the disease, "polyposis coli tumour" tagged with I (inside) tags, and everything else tagged as O (outside).  Model configurationOur Named Entity Recognition model is comprised of the pretrained [BERT](https://arxiv.org/pdf/1810.04805.pdf) model followed by a Token Classification layer.The model is defined in a config file which declares multiple important sections. They are:- **model**: All arguments that are related to the Model - language model, token classifier, optimizer and schedulers, datasets and any other related information- **trainer**: Any argument to be passed to PyTorch Lightning<jupyter_code>WORK_DIR = "WORK_DIR"
os.makedirs(WORK_DIR, exist_ok=True)
MODEL_CONFIG = "token_classification_config.yaml"
# download the model's configuration file 
config_dir = WORK_DIR + '/configs/'
os.makedirs(config_dir, exist_ok=True)
if not os.path.exists(config_dir + MODEL_CONFIG):
    print('Downloading config file...')
    wget.download(f'https://raw.githubusercontent.com/NVIDIA/NeMo/{BRANCH}/examples/nlp/token_classification/conf/' + MODEL_CONFIG, config_dir)
else:
    print ('config file is already exists')
# this line will print the entire config of the model
config_path = f'{WORK_DIR}/configs/{MODEL_CONFIG}'
print(config_path)
config = OmegaConf.load(config_path)
# Note: these are small batch-sizes - increase as appropriate to available GPU capacity
config.model.train_ds.batch_size=8
config.model.validation_ds.batch_size=8
print(OmegaConf.to_yaml(config))<jupyter_output><empty_output><jupyter_text>Model Training Setting up Data within the configAmong other things, the config file contains dictionaries called dataset, train_ds and validation_ds. These are configurations used to setup the Dataset and DataLoaders of the corresponding config.We assume that both training and evaluation files are located in the same directory, and use the default names mentioned during the data download step. So, to start model training, we simply need to specify `model.dataset.data_dir`, like we are going to do below.Also notice that some config lines, including `model.dataset.data_dir`, have `???` in place of paths, this means that values for these fields are required to be specified by the user.Let's now add the data directory path to the config.<jupyter_code># in this tutorial train and dev datasets are located in the same folder, so it is enought to add the path of the data directory to the config
config.model.dataset.data_dir = os.path.join(DATA_DIR, 'NER')

# if you want to decrease the size of your datasets, uncomment the lines below:
# NUM_SAMPLES = 1000
# config.model.train_ds.num_samples = NUM_SAMPLES
# config.model.validation_ds.num_samples = NUM_SAMPLES<jupyter_output><empty_output><jupyter_text>Building the PyTorch Lightning TrainerNeMo models are primarily PyTorch Lightning modules - and therefore are entirely compatible with the PyTorch Lightning ecosystem.Let's first instantiate a Trainer object<jupyter_code>print("Trainer config - \n")
print(OmegaConf.to_yaml(config.trainer))
# lets modify some trainer configs
# checks if we have GPU available and uses it
cuda = 1 if torch.cuda.is_available() else 0
config.trainer.gpus = cuda

# for PyTorch Native AMP set precision=16
config.trainer.precision = 16 if torch.cuda.is_available() else 32

# remove distributed training flags
config.trainer.accelerator = 'DDP'

trainer = pl.Trainer(**config.trainer)<jupyter_output><empty_output><jupyter_text>Setting up a NeMo Experiment¶NeMo has an experiment manager that handles logging and checkpointing for us, so let's use it:<jupyter_code>exp_dir = exp_manager(trainer, config.get("exp_manager", None))

# the exp_dir provides a path to the current experiment for easy access
exp_dir = str(exp_dir)
exp_dir
# complete list of supported BERT-like models
print(nemo_nlp.modules.get_pretrained_lm_models_list())

# specify BERT-like model, you want to use
PRETRAINED_BERT_MODEL = "biomegatron-bert-345m-cased"
# add the specified above model parameters to the config
config.model.language_model.pretrained_model_name = PRETRAINED_BERT_MODEL<jupyter_output><empty_output><jupyter_text>Now, we are ready to initialize our model. During the model initialization call, the dataset and data loaders we'll be prepared for training and evaluation.Also, the pretrained BERT model will be downloaded, note it can take up to a few minutes depending on the size of the chosen BERT model.<jupyter_code>model_ner = nemo_nlp.models.TokenClassificationModel(cfg=config.model, trainer=trainer)<jupyter_output><empty_output><jupyter_text>Monitoring training progressOptionally, you can create a Tensorboard visualization to monitor training progress.If you're not using Colab, refer to [https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks](https://www.tensorflow.org/tensorboard/tensorboard_in_notebooks) if you're facing issues with running the cell below.<jupyter_code>try:
    from google import colab
    COLAB_ENV = True
except (ImportError, ModuleNotFoundError):
    COLAB_ENV = False

# Load the TensorBoard notebook extension
if COLAB_ENV:
    %load_ext tensorboard
    %tensorboard --logdir {exp_dir}
else:
    print("To use tensorboard, please use this notebook in a Google Colab environment.")
# start model training
trainer.fit(model_ner)<jupyter_output><empty_output><jupyter_text>InferenceTo see how the model performs, we can run generate prediction similar to the way we did it earlier<jupyter_code># let's first create a subset of our dev data
! head -n 100 {DATA_DIR}/NER/text_dev.txt > {DATA_DIR}/NER/sample_text_dev.txt
! head -n 100 {DATA_DIR}/NER/labels_dev.txt > {DATA_DIR}/NER/sample_labels_dev.txt<jupyter_output><empty_output><jupyter_text>Now, let's generate predictions for the provided text file.If labels file is also specified, the model will evaluate the predictions and plot confusion matrix.<jupyter_code>model_ner.evaluate_from_file(
    text_file=os.path.join(DATA_DIR, 'NER', 'sample_text_dev.txt'),
    labels_file=os.path.join(DATA_DIR, 'NER', 'sample_labels_dev.txt'),
    output_dir=exp_dir,
    add_confusion_matrix=True,
    normalize_confusion_matrix=True,
    batch_size=1
)
# Please check matplotlib version if encountering any error plotting confusion matrix:
# https://stackoverflow.com/questions/63212347/importerror-cannot-import-name-png-from-matplotlib<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07912579342181189, 'fraction_numerical': 0.008006347374495095, 'mean_word_length': 5.491104868913857, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 51, 'https://': 22, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '12624', 'license': 'Apache-2.0', 'n_tokens_mistral': 4238, 'n_tokens_neox': 3927, 'n_words': 1443, 'path': 'tutorials/nlp/Token_Classification-BioMegatron.ipynb', 'repo_name': 'diego-fustes/NeMo', 'size': '13864'}