<jupyter_start><jupyter_text>1. Load data and classifier<jupyter_code># test_dir = "/Volumes/Kalyan_SSD/SMG/disambiguation/objects_131120/test_computing_space/"
# test_dir = "/Volumes/Kalyan_SSD/SMG/disambiguation/objects_131120/test_photographic_aeronautics/"
test_dir = "/Volumes/Kalyan_SSD/SMG/disambiguation/objects_131120/test_art_locomotives_and_rolling_stock/"
test_dir = "/Volumes/Kalyan_SSD/SMG/disambiguation/objects_131120/test_locomotives_and_rolling_stock/"

X, pairs, pids = load_training_data(test_dir)
d = Disambiguator('OBJECT')
d.load_classifier_from_disk("/Volumes/Kalyan_SSD/SMG/disambiguation/objects_131120/clf.pkl")
d.print_tree(feature_names=pids)<jupyter_output>|--- label <= 0.97
|   |--- label <= 0.80
|   |   |--- class: False
|   |--- label >  0.80
|   |   |--- label <= 0.81
|   |   |   |--- class: True
|   |   |--- label >  0.81
|   |   |   |--- label <= 0.81
|   |   |   |   |--- class: False
|   |   |   |--- label >  0.81
|   |   |   |   |--- class: False
|--- label >  0.97
|   |--- P31 <= 0.13
|   |   |--- P31 <= 0.00
|   |   |   |--- class: True
|   |   |--- P31 >  0.00
|   |   |   |--- class: True
|   |--- P31 >  0.13
|   |   |--- P31 <= 0.75
|   |   |   |--- class: True
|   |   |--- P31 >  0.75
|   |   |   |--- class: True<jupyter_text>2. Predict SMG-Wikidata links using classifier<jupyter_code>pairs_pred = d.get_predictions_table(X, pairs, threshold=0.9)
pairs_pred.sort_values('y_pred_proba', ascending=False).head()<jupyter_output><empty_output><jupyter_text>3. Filter results<jupyter_code># first get only the positive predictions
pairs_pred_positive = pairs_pred[pairs_pred['y_pred'] == True]

len(pairs_pred_positive)
pairs_pred_positive_filtered = filter_cased_wikidata_labels(pairs_pred_positive)
pairs_pred_positive_filtered = remove_wikidata_items_with_min_claims(pairs_pred_positive_filtered, 1)
pairs_pred_positive_filtered = filter_max_wikidata_links(pairs_pred_positive_filtered)

print(f"{len(pairs_pred_positive_filtered)} matches after filtering")
pairs_pred_positive_filtered.sort_values('y_pred_proba', ascending=False).head(20)<jupyter_output><empty_output><jupyter_text>4. Export results<jupyter_code>export_path = test_dir + "preds_positive.csv"
print(export_path)
pairs_pred_positive_filtered.to_csv(export_path)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1518166089965398, 'fraction_numerical': 0.04022491349480969, 'mean_word_length': 4.2929061784897025, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '644168', 'license': 'MIT', 'n_tokens_mistral': 994, 'n_tokens_neox': 907, 'n_words': 106, 'path': 'demos/Record Disambiguation - Using Imported Classifier.ipynb', 'repo_name': 'TheScienceMuseum/heritage-connector', 'size': '2312'}