<jupyter_start><jupyter_text>AUC HeatmapThe heatmap below shows the three AUC-based metrics for two models. Each column is labeled with "MODEL\_NAME"_"METRIC\_NAME"Metrics:* Subgroup AUC: AUC of examples within the identity subgroup.* Negative Cross AUC: AUC of negative (out of class, i.e. non-toxic) examples in the identity subgroup, and positive examples outside the identity subgroup.* Positive Cross AUC: AUC of negative (out of class, i.e. non-toxic) examples outside the identity subgroup, and positive examples in the identity subgroup.Values should range between 0.5-1.0 and higher is better.<jupyter_code>model_bias_analysis.plot_auc_heatmap(madlibs_results, models)<jupyter_output><empty_output><jupyter_text>AEG HeatmapThe heatmap below shows the two Average Equality Gap metrics for two models.Metrics:* Negative AEG: Measures the difference between the distributions of out-of-class examples within the subgroup and outside the subgroup.* Positive AEG: Measures the difference between the distributions of in-class examples within the subgroup and outside the subgroup.0 is the ideal for this metric. Postive values indicate a skew towards higher scores, negative values indicate a skew towards lower scores.<jupyter_code>model_bias_analysis.plot_aeg_heatmap(madlibs_results, models)
model_bias_analysis.plot_auc_heatmap(real_data_results, real_data_models)
model_bias_analysis.plot_aeg_heatmap(real_data_results, real_data_models)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.056910569105691054, 'fraction_numerical': 0.0033875338753387536, 'mean_word_length': 6.898395721925134, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '63776', 'license': 'Apache-2.0', 'n_tokens_mistral': 424, 'n_tokens_neox': 385, 'n_words': 158, 'path': 'unintended_ml_bias/metric_heatmap_example.ipynb', 'repo_name': 'g8a9/unintended-ml-bias-analysis', 'size': '1476'}