<jupyter_start><jupyter_text>Evaluate Presidio AnalyzerThis notebook runs the PresidioAnalyzerEvaluator class on top of synthetic data.One can perform the following changes:1. Replace the synthetic data creation with real data or with other type of synthetic data2. Adapt the Presidio `AnalyzerEngine` to a specific engine with a different set of recognizers or configured to be used on different languages  A. Read dataset for evaluation<jupyter_code>input_samples = read_synth_dataset("../data/synth_dataset.txt")
print("Read {} samples".format(len(input_samples)))
input_samples[0]<jupyter_output><empty_output><jupyter_text>B. Descriptive statistics<jupyter_code>flatten = lambda l: [item for sublist in l for item in sublist]

count_per_entity = Counter([span.entity_type for span in flatten([input_sample.spans for input_sample in input_samples])])
count_per_entity<jupyter_output><empty_output><jupyter_text>C. Match the dataset's entity names with Presidio's entity names<jupyter_code>presidio_entities_map = {
    "PERSON": "PERSON",
    "EMAIL_ADDRESS": "EMAIL_ADDRESS",
    "CREDIT_CARD": "CREDIT_CARD",
    "FIRST_NAME": "PERSON",
    "PHONE_NUMBER": "PHONE_NUMBER",
    "BIRTHDAY": "DATE_TIME",
    "DATE_TIME": "DATE_TIME",
    "DOMAIN": "DOMAIN",
    "CITY": "LOCATION",
    "ADDRESS": "LOCATION",
    "NATIONALITY": "LOCATION",
    "LOCATION": "LOCATION",
    "IBAN": "IBAN_CODE",
    "URL": "DOMAIN_NAME",
    "US_SSN": "US_SSN",
    "IP_ADDRESS": "IP_ADDRESS",
    "ORGANIZATION": "ORG",
    "TITLE" : "O", # skipping evaluation of titles
    "O": "O",
}

new_list = Evaluator.align_entity_types(input_samples, presidio_entities_map)<jupyter_output><empty_output><jupyter_text>D. Recalculate statistics on updated dataset<jupyter_code>## recheck counter
count_per_entity_new = Counter([span.entity_type for span in flatten([input_sample.spans for input_sample in new_list])])
count_per_entity_new<jupyter_output><empty_output><jupyter_text>E. Run the presidio-evaluator framework with Presidio's API as the 'model' at test<jupyter_code>presidio = PresidioAnalyzerWrapper(entities_to_keep=list(count_per_entity_new.keys()))
evaluator = Evaluator(model=presidio)
evaluted_samples = evaluator.evaluate_all(new_list[:100])<jupyter_output><empty_output><jupyter_text>F. Extract statistics- Presicion, recall and F measure are calculated based on a PII/Not PII binary classification per token.- Specific entity recall and precision are calculated on the specific PII entity level.<jupyter_code>evaluation_result = evaluator.calculate_score(evaluted_samples)
evaluation_result.print()<jupyter_output><empty_output><jupyter_text>G. Analyze wrong predictions<jupyter_code>errors = evaluation_result.model_errors
ModelError.most_common_fp_tokens(errors,n=5)
fps_df = ModelError.get_fps_dataframe(errors,entity='PERSON')
if fps_df is not None:
    fps_df[['full_text','token','prediction']]
fns_df = ModelError.get_fns_dataframe(errors,entity='PERSON')
fns_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1068090787716956, 'fraction_numerical': 0.002336448598130841, 'mean_word_length': 7.279005524861878, 'pattern_counts': {'":': 18, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '289416', 'license': 'MIT', 'n_tokens_mistral': 1021, 'n_tokens_neox': 1003, 'n_words': 226, 'path': 'notebooks/Evaluate Presidio Analyzer.ipynb', 'repo_name': 'msebragge/presidio-research', 'size': '2996'}