<jupyter_start><jupyter_text>Pyramid: A Layered Model for Nested Named Entity Recognition - Custom TrainingThis notebook provides a blueprint to train a version of [Pyramid: A Layered Model for Nested Named Entity Recognition](https://www.aclweb.org/anthology/2020.acl-main.525.pdf) on your own data.**It is recommended to use GPU for training.**  Downloading Code<jupyter_code>!git clone https://github.com/federico-giannoni/pyramid-nested-ner.git
!mv pyramid-nested-ner/* . && rm -rf pyramid-nested-ner # move to root
!pip install flair seqeval 2>&1 > /dev/null
from pyramid_nested_ner.model import PyramidNer
from pyramid_nested_ner.data import DataPoint, Entity
from pyramid_nested_ner.modules.word_embeddings.transformer_embeddings import TransformerWordEmbeddings
from pyramid_nested_ner.modules.word_embeddings.pretrained_embeddings import PretrainedWordEmbeddings
from pyramid_nested_ner.data.dataset import PyramidNerDataset
from pyramid_nested_ner.utils.data import rasa_data_reader
from pyramid_nested_ner.training.trainer import PyramidNerTrainer
from pyramid_nested_ner.training.optim import get_default_sgd_optim
from copy import deepcopy

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import torch
import json
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'<jupyter_output><empty_output><jupyter_text>Experiments Setup<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
# TODO: set these paths to point to your data

train_data_path = None  # e.g. "/content/drive/My Drive/your_train_data_path"
test_data_path = None   # e.g. "/content/drive/My Drive/your_test_data_path"
dev_data_path  = None   # e.g "/content/drive/My Drive/your_dev_data_path"
def your_own_data_generator(path):
  """
  Implement this function to yield DataPoint objects
  representing your data. The class definition can be found at: 
  https://github.com/federico-giannoni/pyramid-nested-ner/blob/main/pyramid_nested_ner/data/__init__.py
  """
  # if your data is in rasa json format, you can just uncomment the line below:
  # yield from rasa_data_reader(path)
  pass
pyramid_max_depth = 2  # keep this low if you plan to do inference on CPU (paper uses 15)
train_data = PyramidNerDataset(
  your_own_data_generator(train_data_path), 
  pyramid_max_depth=pyramid_max_depth,
  token_lexicon=lexicon,
  custom_tokenizer=None, 
  char_vectorizer=True,
).get_dataloader(
    shuffle=True,
    batch_size=64,
    device=DEVICE, 
    bucketing=True
)

test_data = PyramidNerDataset(
  your_own_data_generator(test_data_path), 
  pyramid_max_depth=pyramid_max_depth,
  token_lexicon=lexicon,
  custom_tokenizer=None, 
  char_vectorizer=True,
).get_dataloader(
    shuffle=True, 
    batch_size=16,
    device=DEVICE, 
    bucketing=True
)

dev_data = PyramidNerDataset(
  your_own_data_generator(dev_data_path), 
  pyramid_max_depth=pyramid_max_depth,
  token_lexicon=lexicon,
  custom_tokenizer=None, 
  char_vectorizer=True,
).get_dataloader(
    shuffle=True, 
    batch_size=16,
    device=DEVICE, 
    bucketing=True
)<jupyter_output><empty_output><jupyter_text>TrainingFor the `language_model` parameter, you can use any huggingface transformer by specyfing its name. The full list of names is available [here](https://huggingface.co/transformers/pretrained_models.html).If you plan on using an uncased model, you should also pass `language_model_casing=False` to the `PyramidNer` constructor.**Note that using word embeddings from pre-trained language models increases training time (and inference time) by a factor of 10.** During training, embeddings are cached during the first epoch so that the following epochs are faster, but this can not be done during inference. For this reason, you should only use the `language_model` parameter if you're planning on using the model for research purposes.For the `word_embeddings` parameter, you can either provide your own `torch.nn.Embedding` module, or use the names of any of the `WordEmbeddings` from **Flair** that you can find [here](https://github.com/flairNLP/flair/blob/master/flair/embeddings/token.pyL121).<jupyter_code>pyramid_ner = PyramidNer(
  word_lexicon=lexicon,
  entities_lexicon=train_entities,
  word_embeddings=['en-glove', 'en-crawl'],  # 100-dim glove + fasttext
  language_model=None,
  char_embeddings_dim=60,
  encoder_hidden_size=100,
  encoder_output_size=200,
  decoder_hidden_size=100,
  inverse_pyramid=False,
  custom_tokenizer=None,
  pyramid_max_depth=pyramid_max_depth,
  decoder_dropout=0.2,
  encoder_dropout=0.2,
  device=DEVICE,
)

trainer = PyramidNerTrainer(pyramid_ner)
# default optimizer and LR scheduler as described in the paper - feel free to change them.
optimizer, scheduler = get_default_sgd_optim(pyramid_ner.nnet.parameters()) 
ner_model, report = trainer.train(
  train_data, 
  optimizer=optimizer, 
  scheduler=scheduler, 
  restore_weights_on='loss',
  epochs=60, 
  dev_data=dev_data, 
  patience=np.inf, 
  grad_clip=5.0
)
report.plot_loss_report()
report.plot_custom_report('micro_f1')
print(trainer.test_model(test_data, out_dict=False))<jupyter_output><empty_output><jupyter_text>Inference<jupyter_code>out = pyramid_ner.parse("your own test sentence")
print(out)<jupyter_output><empty_output><jupyter_text>Saving for later use<jupyter_code>pyramid_ner.save(path='.', name='pyramid_ner')
!tar -cvzf pyramid_ner  # compress...<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0852598483447383, 'fraction_numerical': 0.008507490290364343, 'mean_word_length': 5.951156812339332, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '651918', 'license': 'MIT', 'n_tokens_mistral': 1891, 'n_tokens_neox': 1802, 'n_words': 432, 'path': 'test/Pyramid Nested NER - Custom Training.ipynb', 'repo_name': 'm-stoeckel/pyramid-nested-ner', 'size': '5407'}