<jupyter_start><jupyter_text>Baseline: Enter Full Mention into Wikipedia2vec get_entities()Our most direct step is to use Wikipedia2vec's API and its get_entities() function to enter `full_mention` directly into that. We do that as our first prediction step and baseline model, with the assumption that a returned result is near a 100% chance of being the correct page. We test this hypothesis at the end of this notebook. Our aim is to set a baseline of "just use the available package" before showing that adding customized work can improve accuracy.  Import Packages<jupyter_code>import os
import time

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Progress bar
from tqdm import tqdm<jupyter_output><empty_output><jupyter_text>Load Processed ACY Input<jupyter_code># Base path to input
acy_path = '../../data/aida-conll-yago-dataset/'

# Load data
acy_input = pd.read_csv(os.path.join(acy_path, "Aida-Conll-Yago-Input.csv"), delimiter=",")
acy_input.head(3)
# Re-name for this predictive step
w2v_predictions = acy_input.copy()<jupyter_output><empty_output><jupyter_text>Import Wikipedia2Vec ModelWe use Wikipedia2Vec's 100D pre-trained embeddings for our search functionality. We experimented with higher dimension embeddings and find no improvement in accuracy.<jupyter_code># Package
from wikipedia2vec import Wikipedia2Vec
%%time
# Load unzipped pkl file with word embeddings
w2v = Wikipedia2Vec.load("../../embeddings/enwiki_20180420_100d.pkl")<jupyter_output>CPU times: user 94.6 ms, sys: 123 ms, total: 218 ms
Wall time: 277 ms<jupyter_text>Query using `full_mention`<jupyter_code># Track success rate for returned values
successes = 0
queries = 0
failed_searches = []
preds_w2v_getentity = []

# Run through each full_mention
for full_mention in tqdm(acy_input['full_mention']):
    
    # Query API
    entity = w2v.get_entity(full_mention)
    
    # Increment count
    queries += 1
    if entity is not None:
        successes += 1
    else:
        # Save X% of random failures to understand trends
        if np.random.uniform() <= 0.1:
            failed_searches.append(full_mention)
    
    # Save just title, not entity object
    try:
        entity = entity.title
    except:
        pass

    # Save prediction
    preds_w2v_getentity.append(entity)
print("Query Success Rate: ", round(successes/queries*100, 3),"%")
# Display random selection of failures
np.random.choice(failed_searches, 50)<jupyter_output><empty_output><jupyter_text>Just plugging in the full mention to Wikipedia2vec receives a 60.298% query success rate, meaning the package is able to return any result at all. Most of the failed queries seem like reasonable, non-special text strings. Since Wikipedia2vec essentially matches on title searching, this reasonably suggests that assuming a full mention is a page title is an unlikely method to achieve great results and that some intermediate step must occur to link natural language to a Wikipedia page title.<jupyter_code># Append predictions to table
w2v_predictions['preds_w2v'] = preds_w2v_getentity
w2v_predictions.head(10)<jupyter_output><empty_output><jupyter_text>Assess Accuracy of Predictions<jupyter_code># Calculate accuracy
print("****************************")
overall_accuracy = (w2v_predictions['preds_w2v'] == w2v_predictions['wikipedia_title']).sum()
print(f"All-Inclusive Predictive Accuracy: {round(overall_accuracy / len(w2v_predictions) * 100, 3)}%")
print("Note: All Full Mentions, whether they have a known true value or not.")

print("****************************")
filtered_df = w2v_predictions[w2v_predictions['wikipedia_page_ID'].notnull()]
mention_accuracy = (filtered_df['preds_w2v'] == filtered_df['wikipedia_title']).sum()
print(f"Filter Empty Mentions Predictive Accuracy: {round(mention_accuracy / len(filtered_df) * 100, 3)}%")
print("Note: Only Full Mentions with a known true value")


print("****************************")
filtered_df = w2v_predictions[(w2v_predictions['wikipedia_page_ID'].notnull())\
                                   & (w2v_predictions['preds_w2v'].notnull())]
result_accuracy = (filtered_df['preds_w2v'] == filtered_df['wikipedia_title']).sum()
print(f"Filter Empty Known Trues & Results Predictive Accuracy: {round(result_accuracy / len(filtered_df) * 100, 3)}%")
print("Note: Only full mentions with a known true value that returned a result from Wikipedia2vec")<jupyter_output>****************************
All-Inclusive Predictive Accuracy: 39.578%
Note: All Full Mentions, whether they have a known true value or not.
****************************
Filter Empty Mentions Predictive Accuracy: 54.228%
Note: Only Full Mentions with a known true value
****************************
Filter Empty Known Trues & Results Predictive Accuracy: 73.809%
Note: Only full mentions with a known true value that returned a result from Wikipedia2vec<jupyter_text>We have chosen to standardize our comparisons across model types on full mentions with known true values. This model achieves an accuracy score of 54.228% with known true values. This comes on a base of ~60% query success, suggesting that a very high proportion of successful queries are accurate (54% / 60% = 90% prediction accuracy for successful queries), but that performance cannot be improved without improving linking to Wikipedia pages.<jupyter_code># Confirm effectiveness of Wikipedia2vec's get_entity() on known true answers (non-null)
# Basically, if we plug in the literal Wikipedia URL, will it return the right result?
none_count = 0
answers = []
wikipedia_title = w2v_predictions[w2v_predictions['wikipedia_title'].notnull()]['wikipedia_title']
for r in tqdm(wikipedia_title):
    answer = w2v.get_entity(r)
    if answer is None:
        none_count += 1
    try:
        answers.append(answer.title)
    except AttributeError:
        answers.append(answer)
print("****************************")
print("Wikipedia2vec failed to return {} searches, {}%".format(none_count, round(none_count/len(wikipedia_title)*100,3)))
accurate_preds = np.array([True if i==j else False for i, j in zip(answers, list(filter(None, wikipedia_title)))]).sum()
print("****************************")
print(f"All-Inclusive Predictive Accuracy: {round(accurate_preds / len(answers)*100, 3)}%")
print("****************************")<jupyter_output>100%|██████████| 15918/15918 [00:00<00:00, 122137.15it/s]<jupyter_text>We perform a second check of Wikipedia2Vec's effectiveness by querying all the known true response values, which represent known Wikipedia page titles. Wikipedia2vec returns 98.579% with high accuracy for known true values. The package is clearly effective at returning title searches; this is just not enough for our problem.  Save predictive dataframe for input to next step<jupyter_code># See final output
w2v_predictions.head(3)
# Save dataframe
preds_path = '../../predictions/'
w2v_predictions.to_csv(os.path.join(preds_path, "w2v_getentities.csv"), index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12312740761877586, 'fraction_numerical': 0.025110572121557997, 'mean_word_length': 5.575984990619137, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '311114', 'license': 'MIT', 'n_tokens_mistral': 2196, 'n_tokens_neox': 2032, 'n_words': 702, 'path': 'notebooks/pipeline/baseline_wikipedia2vec.ipynb', 'repo_name': 'TheDigitalFrontier/entity-disambiguation', 'size': '7009'}