<jupyter_start><jupyter_text>NED Benchmark TutorialIn this tutorial, we demonstrate how to use a pretrained Bootleg NED model to run inference on RSS500 and KORE50, two standard sentence-level NED benchmarks. Requirements To run this tutorial, you'll need to download the following: - Pretrained Bootleg model and config [here](https://bootleg-emb.s3.amazonaws.com/models/2020_10_22/bootleg_wiki.tar.gz)- RSS500 data [here](https://bootleg-emb.s3.amazonaws.com/data/rss500.tar.gz)- KORE50 data [here](https://bootleg-emb.s3.amazonaws.com/data/kore50.tar.gz)- Entity data [here](https://bootleg-emb.s3.amazonaws.com/data/wiki_entity_data.tar.gz)- Embedding data [here](https://bootleg-emb.s3.amazonaws.com/data/emb_data.tar.gz)- Pretrained BERT model [here](https://bootleg-emb.s3.amazonaws.com/pretrained_bert_models.tar.gz)For convenience, you can run the commands below (from the root directory of the repo) to download all the above files and unpack them to `models`, `data`, and `pretrained_bert_models` directories. It will take several minutes to download all the files.     bash download_model.sh     bash download_data.sh     bash download_bert.sh      1. Prepare the Config File Necessary import statements. You will need to have installed bootleg as package to run these (see Installation instructions in the README).<jupyter_code>import sys
import logging
from importlib import reload
reload(logging)
logging.basicConfig(stream=sys.stdout, format='%(asctime)s %(message)s', level=logging.INFO)
logger = logging.getLogger(__name__)

from bootleg import run
from bootleg.utils.parser_utils import get_full_config<jupyter_output><empty_output><jupyter_text>If you have a GPU with at least 12GB of memory available, set the below to `False` to run inference on a GPU.<jupyter_code>use_cpu = True<jupyter_output><empty_output><jupyter_text>Load the model config so we can set additional parameters and load the saved model during evaluation.<jupyter_code>root_dir = # FILL IN FULL PATH TO ROOT REPO DIRECTORY HERE
config_path = f'{root_dir}/models/bootleg_wiki/bootleg_config.json'
config_args = get_full_config(config_path)<jupyter_output><empty_output><jupyter_text>Update the config parameters to point to the downloaded model checkpoint and data.<jupyter_code># set the model checkpoint path 
config_args.run_config.init_checkpoint = f'{root_dir}/models/bootleg_wiki/bootleg_model.pt'

# set the path for the entity db and candidate map
config_args.data_config.entity_dir = f'{root_dir}/data/wiki_entity_data'
config_args.data_config.alias_cand_map = 'alias2qids_rss500.json'

# set the data path and RSS500 test file 
config_args.data_config.data_dir = f'{root_dir}/data/rss500'
config_args.data_config.test_dataset.file = 'test_rss500.jsonl'

# set the embedding paths 
config_args.data_config.emb_dir =  f'{root_dir}/data/emb_data'
config_args.data_config.word_embedding.cache_dir =  f'{root_dir}/pretrained_bert_models'

# set the save directory 
config_args.run_config.save_dir = f'{root_dir}/results'

# set whether to run inference on the CPU
config_args.run_config.cpu = use_cpu<jupyter_output><empty_output><jupyter_text>2. Run Inference for RSS500 Once the config is set up, run model evaluation! You should get that 428/450 mentions (men) are correct (crct).<jupyter_code>run.model_eval(args=config_args, mode="eval", logger=logger, is_writer=True)<jupyter_output>2020-10-21 17:19:19,476 PyTorch version 1.5.0 available.<jupyter_text>The `final_loss` head corresponds to the final prediction head. We only have a single data subset, or slice, for the benchmark, which is the overall slice that includes all mentions (we call this slice `final_loss`). The `f1_pop` metric is the score on simple baseline which simply predicts most popular candidate for each mention without any other contextual information. We see that Bootleg improves over 29 F1 points over this baseline. The F1 score reported here is the micro-averaged F1 score over the entities and assumes 100% candidate recall (every mention has a candidate list). However, for the benchmarks, some mentions are in the benchmark but do not have a corresponding candidate list. Thus, for benchmarks we need to re-compute the F1 taking into account the candidate recall, where the number above is equivalent to the benchmark precision.   3. Analyze the ErrorsTo understand what examples Bootleg gets wrong, we also support a `dump_preds` mode. Rather than computing aggregate quality metrics, this mode writes a jsonlines file with the predicted candidates for each mention and their associated probabilities. Running this is very similar as before, except we need to switch the mode to `dump_preds`.<jupyter_code>pred_file, _ = run.model_eval(args=config_args, mode="dump_preds", logger=logger, is_writer=True)<jupyter_output>2020-10-21 17:20:51,382 Loading entity_symbols...
2020-10-21 17:20:57,033 Loaded entity_symbols with 5310039 entities.
2020-10-21 17:20:57,038 Loading slices...
2020-10-21 17:20:57,043 Finished loading slices.
2020-10-21 17:21:15,805 Loading dataset...
2020-10-21 17:21:15,818 Finished loading dataset.
2020-10-21 17:21:22,605 Loading embeddings...
2020-10-21 17:22:00,844 Finished loading embeddings.
2020-10-21 17:22:01,200 Loading model from /dfs/scratch0/lorr1/bootleg/bootleg-internal/new_tutorial_data/models/bootleg_wiki/bootleg_model.pt...
2020-10-21 17:22:13,147 Successfully loaded model from /dfs/scratch0/lorr1/bootleg/bootleg-internal/new_tutorial_data/models/bootleg_wiki/bootleg_model.pt starting from checkpoint epoch 1 and step 0.
2020-10-21 17:22:13,259 ************************DUMPING PREDICTIONS FOR test_rss500.jsonl************************
2020-10-21 17:22:13,361 368 samples, 23 batches, 353 len dataset
2020-10-21 17:26:29,812 Writing predictions to /dfs/scratch0/lorr1/bootle[...]<jupyter_text>We provide utility functions to load in the predicted labels as well as the original file and generate a merged [pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/user_guide/10min.html) with both predicted and gold labels.<jupyter_code>from utils import score_predictions, load_title_map, load_cand_map
import pandas as pd
pd.options.display.max_colwidth = 500

title_map = load_title_map(f'{root_dir}/data/wiki_entity_data/entity_mappings')
pred_df = score_predictions(orig_file=f'{root_dir}/data/rss500/test_rss500.jsonl', 
                 pred_file=pred_file,
                 title_map=title_map)<jupyter_output>100%|██████████| 353/353 [00:00<00:00, 7404.76it/s]<jupyter_text>Let's take a look at the DataFrame! The schema returned is:`sentence`: the sentence,`sent_idx`: the unique index of the sentences,`aliases`: all mentions in the sentence,`span`: the span of the predicted mention,`slices`: the slices this example is in (see the training tutorials for definitions),`alias`: the mention to predict,`alias_idx`: the index of the mention (the first mention, second mentions, ...),`is_gold_label`: is the mention a weak label or a gold label from the data,`gold_qid`: the gold entity,`pred_qid`: the predicted entity,`gold_title`: the gold title,`pred_title`: the predicted title,`all_gold_qids`: all gold entities in the sentence,`all_pred_qids`: all predicted entities,`gold_label_aliases`: all gold mentions in the sentence,`all_is_gold_labels`: which mentions are gold or weak labels,`all_spans`: the spans of all mentions<jupyter_code>pred_df.sample(10)<jupyter_output><empty_output><jupyter_text>You can also add the possible candidates with their associated socres by passing the `cand_map` arg to `score_predictions`.<jupyter_code># Load candidate mappings to pass as input
cands_map = load_cand_map(f'{root_dir}/data/wiki_entity_data/entity_mappings', 'alias2qids_rss500.json')
pred_df = score_predictions(orig_file=f'{root_dir}/data/rss500/test_rss500.jsonl',
                 pred_file=pred_file,
                 title_map=title_map,
                 cands_map=cands_map)

pred_df.sample(10)<jupyter_output><empty_output><jupyter_text>We can write functions over the DataFrame to help with error analysis. For instance, to get all incorrect examples, we use the below command.<jupyter_code>pred_df[pred_df['gold_qid'] != pred_df['pred_qid']].sample(10)<jupyter_output><empty_output><jupyter_text>4. Repeat for KORE50 Use the following config to run eval for KORE50.<jupyter_code># set the model checkpoint path 
config_args.run_config.init_checkpoint = f'{root_dir}/models/bootleg_wiki/bootleg_model.pt'

# set the path for the entity db and candidate map
config_args.data_config.entity_dir = f'{root_dir}/data/wiki_entity_data'
config_args.data_config.alias_cand_map = 'alias2qids_kore50.json'

# set the data path and RSS500 test file 
config_args.data_config.data_dir = f'{root_dir}/data/kore50'
config_args.data_config.test_dataset.file = 'test_kore50.jsonl'

# set the embedding paths 
config_args.data_config.emb_dir =  f'{root_dir}/data/emb_data'
config_args.data_config.word_embedding.cache_dir =  f'{root_dir}/pretrained_bert_models'

# set the save directory 
config_args.run_config.save_dir = f'{root_dir}/results'

# set whether to run inference on the CPU
config_args.run_config.cpu = use_cpu

run.model_eval(args=config_args, mode="eval", logger=logger, is_writer=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09525350593311759, 'fraction_numerical': 0.042934196332254586, 'mean_word_length': 5.928998505231689, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 42, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '624552', 'license': 'Apache-2.0', 'n_tokens_mistral': 3242, 'n_tokens_neox': 2896, 'n_words': 864, 'path': 'tutorials/benchmark_tutorial.ipynb', 'repo_name': 'mleszczy/bootleg', 'size': '9270'}