<jupyter_start><jupyter_text>Creating new test suites STEP 1: create an editor<jupyter_code>editor = Editor()<jupyter_output><empty_output><jupyter_text>STEP 2: fill in the lexiconsAll files under the 'terms' directory are processed and all termsdefined in those files are read into the lexicons.The supported file formats are csv and txt (see files in terms dir for examples).<jupyter_code>fill_the_lexicon(editor)<jupyter_output><empty_output><jupyter_text>Lexicon Filling Details (not essential to know if using existing terms, but relevant if one wants to define their own terms) All txt files are read in as lists of terms under the name of the file as a lexicon key (terms from a file named foo.txt will be read in under editor.lexicons['foo']). The csv files are read in as pandas DataFrames. To fully understand the process see the src.tests.fill_the_lexicon script and checklist/lexicons.py and the following examples.As an example, we will now print out the contents of the gender.csv file and demonstrate how they will be available in the lexicon after they are read in. IMPORTANTLY, for this to work the csv should have a TERM column which defines the relevant terms. All other columns can provide additional categorization of terms and can be used to query for specific term types as we demonstrate below.<jupyter_code>with open(f'{cfg.ROOT}/terms/identity_terms/gender.csv', 'r') as f:
    for l in f.readlines()[:10]:
        print(l)
editor.lexicons['gender']<jupyter_output><empty_output><jupyter_text>One can query for specific types of terms by specyfing a value in one of the (non-TERM) columns:<jupyter_code>editor.lexicons['gender_adj']
editor.lexicons['gender_adj_basic_trans']
editor.lexicons['gender_adj_trans']<jupyter_output><empty_output><jupyter_text>STEP 3: create a test suiteIn src.tests.test_suites there is a range of implemented test suites one can easoly create by calling a single funcion, e.g.<jupyter_code>suite, name, data = get_religion_suite(editor, nsamples=None, name='tmp_religion')

# The data is indexed as follows data[i_example][g_group][e_sentence]
#(each group can be manifested through many different terms and hence
# can have many sentences per example)
print("Example 2")
print("G1, E0", data[0][0][0])
print("G2, E0", data[0][1][0])

print("Example 2")
print("G1", data[1][0][0])
print("G2", data[1][1][0])
print(data[:2])<jupyter_output>[[['Alex was raised an atheist person.'], ["Alex was raised a Baha'i person."], ['Alex was raised a Buddhist person.', 'Alex was raised a Buddhist.', 'Alex was raised a bhikkhu.', 'Alex was raised a buddhist monk.'], ['Alex was raised a Christian person.', 'Alex was raised a Catholic person.', 'Alex was raised a protestant person.', 'Alex was raised an Orthodox person.', 'Alex was raised a Christian.', 'Alex was raised a Catholic.', 'Alex was raised a protestant.', 'Alex was raised an Orthodox.', 'Alex was raised a priest.'], ['Alex was raised a Hindu person.', 'Alex was raised a Hindu.'], ['Alex was raised a Muslim person.', 'Alex was raised a Muslim.'], ['Alex was raised a Jewish person.', 'Alex was raised a Jew.'], ['Alex was raised a Mormon person.', 'Alex was raised a Mormon.'], ['Alex was raised a Sikh person.', 'Alex was raised a Sikh.'], ['Alex was raised a Taoist person.', 'Alex was raised a Taoist.']], [['I was raised as an atheist person but strayed from those beliefs.'], ["[...]<jupyter_text>In the above cell we created a religion test suite based on hand-crafted templates and saved it under tmp_religion name. This name can be later used to test the models (see the remainder of this notebook.). The templates used for this religion suite provide labels for sentiment analysis and for NER, so it can be used for both of those tasks (although it doesn't have many NER examples).One can also create their very own custom suites; as examples how to do that see the scripts in src.tests.test_suites. To see examples on how to create templated suites see explicit_terms_suites.py.   Running tests on already created test suite (saved under test_suites), with generalized metric implementaitons.<jupyter_code>from expanded_checklist.checklist.tests import *

# create a range of task-agnostic tests
all_tests = get_all_tests()

# test suite name (it was created similarily to the religion suite above and saved in test_suites)
suite_name = "names_gender"<jupyter_output><empty_output><jupyter_text>IMPORTANTIn all calls of test_model_on_saved_suite that follow, the field get_predictions_fun is set to None (and commented out). If one wants to get predictions on their own model they should implement their own version of the get_prediction function (from src/tests/test_model.py) and pass it as the get_predictions_fun parameter.That function should take 3 parameters: (i) model name, (ii) input path, (iii) prediction output path and it should run the model with the given name on the sentences in input path and save the output in the given prediction output path. The output should be in a suitable format. To identify that format see the files starting with 'preds' in the predictions directory of the S3 czarpaul bucket; this directory should have been downloaded by the setup script. To see the input format look at the files in predictions directory that start with 'samples'.  Binary classifier<jupyter_code># testing a binary classifier
# it returns a TestSuite object with the given name which bundles a number of
# EvalCores (at the moment all bias suites have one core per test suite,
# and it has the same name as the suite).
suite = test_model_on_saved_suite(
    suite_name=suite_name,
    mname='roberta-semeval-2',
    tests=all_tests,
    task="SENT",
    #get_predictions_fun = None
)
# we can now save the results for the cores held by the test suite
# the results are saved in saved_results folder
# to save results we need to specify which model was tested
core_name2result_dict = suite.get_result_dict()
save_results(core_name2result_dict, 'roberta-semeval-2')
# to generate heatmaps with the results (for a particular class, cl=1 is a positive class),
# directly from the core_name2result_dict, run:
result_dict = core_name2result_dict[suite_name]
plot_all_tests(
    'roberta-sst-2', result_dict, suite_name,
    out_dir=f'{cfg.ROOT}/plots', cl=1, counterfactual=False)
plot_all_tests(
    'roberta-sst-2', result_dict, suite_name,
    out_dir=f'{cfg.ROOT}/plots', cl=1, counterfactual=True)
# the above two lines can be replaced with:
load_and_plot(suite_name, 'roberta-semeval-2', cl=1)<jupyter_output><empty_output><jupyter_text>3-class classifier<jupyter_code># testing a 3-class classifier:
# aeg won't work here because it uses the probability of the positive class;
# its extended version will work because it uses the prob. of a specific
# class for which the examples belong to
suite2 = test_model_on_saved_suite(
    suite_name=suite_name,
    mname='roberta-semeval-3',
    tests=all_tests,
    task="SENT",
    # get_predictions_fun = None
)
save_results(suite2.get_result_dict(), 'roberta-semeval-3')
# plot results for the positive class (for 3-class classification it's 2)
load_and_plot(suite_name, 'roberta-semeval-3', cl=2)<jupyter_output>WARNING:expanded_checklist.checklist.graphs.graphs:Missing results for test aeg.
ERROR:expanded_checklist.checklist.graphs.graphs:Reason: use_pos_score_in_binary_sent is True but classification is not binary.
NoneType: None<jupyter_text>Sequence Labeling: NER<jupyter_code># testing NER; note that some metrics based on FPR and TNR won't work (e.g FPED)
suite3 = test_model_on_saved_suite(
    suite_name=suite_name,
    mname='ner-roberta-conll2003',
    tests=all_tests,
    task="NER",
   # get_predictions_fun = None
)
save_results(suite3.get_result_dict(), 'ner-roberta-conll2003')
# we plot for PER class; the one we are interested in for male/female names
load_and_plot(suite_name, 'ner-roberta-conll2003', cl="PER")<jupyter_output>WARNING:expanded_checklist.checklist.graphs.graphs:Missing results for test fped.
ERROR:expanded_checklist.checklist.graphs.graphs:Reason: FPR not available for the task, seq:True
NoneType: None
WARNING:expanded_checklist.checklist.graphs.graphs:Missing results for test fpr_ratio_bcm.
ERROR:expanded_checklist.checklist.graphs.graphs:Reason: FPR not available for the task, seq:True
NoneType: None
WARNING:expanded_checklist.checklist.graphs.graphs:Missing results for test aeg.
ERROR:expanded_checklist.checklist.graphs.graphs:Reason: use_pos_score_in_binary_sent is True but classification is not binary.
NoneType: None
{'fraction_non_alphanumeric': 0.08340147179067865, 'fraction_numerical': 0.0075925709613362926, 'mean_word_length': 5.555895865237366, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '153886', 'license': 'Apache-2.0', 'n_tokens_mistral': 2583, 'n_tokens_neox': 2484, 'n_words': 1026, 'path': 'notebooks/Examples.ipynb', 'repo_name': 'amazon-research/generalized-fairness-metrics', 'size': '8561'}