<jupyter_start><jupyter_text>Evaluate your Presidio instance via the Presidio API  A. Read dataset for evaluation<jupyter_code>input_samples = read_synth_dataset("../data/synth_dataset.txt")
print("Read {} samples".format(len(input_samples)))<jupyter_output><empty_output><jupyter_text>B. Descriptive statistics<jupyter_code>flatten = lambda l: [item for sublist in l for item in sublist]

count_per_entity = Counter([span.entity_type for span in flatten([input_sample.spans for input_sample in input_samples])])
count_per_entity<jupyter_output><empty_output><jupyter_text>C. Match the dataset's entity names with Presidio's entity names<jupyter_code># Mapping between dataset entities and Presidio entities. Key: Dataset entity, Value: Presidio entity
entities_mapping = {
    'PERSON': 'PERSON',
    'EMAIL': 'EMAIL_ADDRESS',
    'CREDIT_CARD': 'CREDIT_CARD',
    'FIRST_NAME': 'PERSON',
    'PHONE_NUMBER': 'PHONE_NUMBER',
    'LOCATION':'LOCATION',
    # 'BIRTHDAY': 'DATE_TIME',
    # 'DATE': 'DATE_TIME',
    'DOMAIN': 'DOMAIN',
    #    'CITY': 'LOCATION',
    #    'ADDRESS': 'LOCATION',
    'IBAN': 'IBAN_CODE',
    # 'URL': 'DOMAIN_NAME',
    'US_SSN': 'US_SSN',
    'IP_ADDRESS': 'IP_ADDRESS',
    # 'ORGANIZATION':'ORG'
    'O': 'O'
}
presidio_fields = ['CREDIT_CARD', 'CRYPTO', 'DATE_TIME', 'DOMAIN_NAME', 'EMAIL_ADDRESS', 'IBAN_CODE',
                           'IP_ADDRESS', 'NRP', 'LOCATION', 'PERSON', 'PHONE_NUMBER', 'US_SSN']

new_list = ModelEvaluator.align_input_samples_to_presidio_analyzer(input_samples,
                                                                   entities_mapping,
                                                                   presidio_fields)<jupyter_output><empty_output><jupyter_text>D. Recalculate statistics on updated dataset<jupyter_code>## recheck counter
count_per_entity_new = Counter([span.entity_type for span in flatten([input_sample.spans for input_sample in new_list])])
count_per_entity_new<jupyter_output><empty_output><jupyter_text>E. Run the presidio-evaluator framework with Presidio's API as the 'model' at test<jupyter_code>from presidio_evaluator import PresidioAPIEvaluator
presidio = PresidioAPIEvaluator(entities_to_keep=list(count_per_entity_new.keys()),endpoint=MY_PRESIDIO_ENDPOINT)
evaluted_samples = presidio.evaluate_all(new_list[:100])<jupyter_output><empty_output><jupyter_text>F. Extract statistics- Presicion, recall and F measure are calculated based on a PII/Not PII binary classification per token.- Specific entity recall and precision are calculated on the specific PII entity level.<jupyter_code>evaluation_result = presidio.calculate_score(evaluted_samples)
evaluation_result.print()<jupyter_output><empty_output><jupyter_text>G. Analyze wrong predictions<jupyter_code>errors = evaluation_result.model_errors
ModelEvaluator.most_common_fp_tokens(errors,n=5)
fps_df = ModelEvaluator.get_fps_dataframe(errors,entity='PERSON')
fps_df[['full_text','token','prediction']]
fns_df = ModelEvaluator.get_fns_dataframe(errors,entity='PERSON')
fns_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11223491027732463, 'fraction_numerical': 0.0013050570962479609, 'mean_word_length': 5.308641975308642, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '286223', 'license': 'MIT', 'n_tokens_mistral': 1038, 'n_tokens_neox': 1025, 'n_words': 193, 'path': 'notebooks/Evaluate Presidio-API.ipynb', 'repo_name': 'diwu1989/presidio-research', 'size': '3065'}