<jupyter_start><jupyter_text>Example: generating implications<jupyter_code>gen = qa_consistency.implication.ImplicationsSquad()
passage = 'Kublai originally named his eldest son, Zhenjin, as the Crown Prince, \
but he died before Kublai in 1285.'
gen.implications('When did Zhenjin die?', '1285', passage)<jupyter_output>Your label namespace was 'pos'. We recommend you use a namespace ending with 'labels' or 'tags', so we don't add UNK and PAD tokens by default to your vocabulary.  See documentation for `non_padded_namespaces` parameter in Vocabulary.<jupyter_text>Generating implications for all possible squad answers on dev dataset. You can skip this and load my precomputed implications below.<jupyter_code>data, answers = qa_consistency.dataset_utils.load_squad('/home/marcotcr/datasets/squad/')
answer_texts = [[x[0] for x in y] for y in answers]
questions = [x['question'].strip() for x in data]
passages = [x['passage'].strip() for x in data]
all_qs, all_as, all_contexts = qa_consistency.dataset_utils.question_answers_context_product(questions, answer_texts, passages)
parsed_dataset = gen.parse_dataset(all_qs, all_as, all_contexts)
implications = [gen.implications_from_parsed(x) for x in parsed_dataset]
idxs = np.random.choice(len(implications), 5, replace=False)
for i in idxs:
    if not implications[i]:
        continue
    print(i, parsed_dataset[i])
    for x in implications[i]:
        print(x)
    print()
output_folder = '/home/marcotcr/tmp/'
squad_path = '/home/marcotcr/datasets/squad/dev-v1.1.json'
all_imps = {}
for qa, imp in zip(parsed_dataset, implications):
    all_imps[qa.as_tuple()] = imp
pickle.dump(all_imps, open(os.path.join(output_folder, "squad_imps.pkl"), 'wb'))<jupyter_output><empty_output><jupyter_text>Start from here if you want to use precomputed implications (link to pkl file in the repository's README)<jupyter_code>output_folder = '/home/marcotcr/tmp/'
squad_path = '/home/marcotcr/datasets/squad/dev-v1.1.json'
all_imps = pickle.load(open(os.path.join(output_folder, "squad_imps.pkl"), 'rb'))
squad_jsonl_path = os.path.join(output_folder, 'dev_allennlp.jsonl')
squad_preds_jsonl = os.path.join(output_folder, 'squad_dev_preds.jsonl')
squad_preds_json = os.path.join(output_folder, 'squad_dev_preds.json')
squad_consistency_path  = os.path.join(output_folder, 'squad_consistency.json')
squad_consistency_jsonl  = os.path.join(output_folder, 'squad_consistency.jsonl')
squad_consistency_preds_jsonl = os.path.join(output_folder, 'squad_consistency_preds.jsonl')
squad_consistency_preds_json = os.path.join(output_folder, 'squad_consistency_preds.json')<jupyter_output><empty_output><jupyter_text>We are using allennlp, so we need to convert the original dataset to a jsonl file before getting predictions<jupyter_code>qa_consistency.dataset_utils.squad_to_allennlp(squad_path, squad_jsonl_path)
print('Run:')
print('allennlp predict https://s3-us-west-2.amazonaws.com/allennlp/models/bidaf-model-2017.09.15-charpad.tar.gz\
 %s --output-file /home/marcotcr/tmp/%s --cuda-device 0 --silent' % (squad_jsonl_path, squad_preds_jsonl))<jupyter_output>Run:
allennlp predict https://s3-us-west-2.amazonaws.com/allennlp/models/bidaf-model-2017.09.15-charpad.tar.gz /home/marcotcr/tmp/dev_allennlp.jsonl --output-file /home/marcotcr/tmp//home/marcotcr/tmp/squad_dev_preds.jsonl --cuda-device 0 --silent<jupyter_text>Transform allennlp output to the official squad output format:<jupyter_code>qa_consistency.dataset_utils.allennlp_preds_to_squad_format(squad_jsonl_path, squad_preds_jsonl, squad_preds_json)<jupyter_output><empty_output><jupyter_text>Generate a consistency dataset to check exact match predictions:<jupyter_code>qa_consistency.dataset_utils.generate_implication_squad(squad_path,
                                                        squad_preds_json,
                                                        all_imps,
                                                        squad_consistency_path)<jupyter_output><empty_output><jupyter_text>Convert datset to jsonl (for allennlp):<jupyter_code>qa_consistency.dataset_utils.squad_to_allennlp(squad_consistency_path, squad_consistency_jsonl)
print('Run:')
print('allennlp predict https://s3-us-west-2.amazonaws.com/allennlp/models/bidaf-model-2017.09.15-charpad.tar.gz\
 %s --output-file /home/marcotcr/tmp/%s --cuda-device 0 --silent' % (squad_consistency_jsonl, squad_consistency_preds_jsonl))<jupyter_output>Run:
allennlp predict https://s3-us-west-2.amazonaws.com/allennlp/models/bidaf-model-2017.09.15-charpad.tar.gz /home/marcotcr/tmp/squad_consistency.jsonl --output-file /home/marcotcr/tmp//home/marcotcr/tmp/squad_consistency_preds.jsonl --cuda-device 0 --silent<jupyter_text>Convert allennlp output to squad format:<jupyter_code>qa_consistency.dataset_utils.allennlp_preds_to_squad_format(squad_consistency_jsonl, squad_consistency_preds_jsonl, squad_consistency_preds_json)<jupyter_output><empty_output><jupyter_text>Evaluate consistency:<jupyter_code>stats = qa_consistency.dataset_utils.evaluate_consistency_squad(squad_consistency_path, squad_consistency_preds_json)
print('Consistency by implication type:')
print()
for x, v in stats.items():
    if x == 'all':
        continue
    print('%s : %.1f' % (x, 100* v))
print()
print('Avg  : %.1f' % (100 * stats['all']))<jupyter_output>Consistency by implication type:

prep : 74.0
dobj : 68.0
subj : 70.5
amod : 75.1

Avg  : 72.9
{'fraction_non_alphanumeric': 0.11417468541820873, 'fraction_numerical': 0.014988897113249446, 'mean_word_length': 5.388888888888889, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '630962', 'license': 'MIT', 'n_tokens_mistral': 2018, 'n_tokens_neox': 1923, 'n_words': 340, 'path': 'notebooks/SQuAD.ipynb', 'repo_name': 'marcotcr/qa_consistency', 'size': '5404'}