<jupyter_start><jupyter_text>---- Tutorial for Evaluating Fairness in Binary Classification ---- Evaluating Your Own Model? Looking for a Template? **Please note: This notebook is intended as an introduction to and reference for evaluating the fairness of a set of machine learning models using FairMLHealth, and it is not designed to work with outside models.**  We recommend using our [Template-BinaryClassificationAssessment](../templates/Template-BinaryClassificationAssessment.ipynb), which also has a useful example: [ Example-BinaryClassificationTemplate]( Example-BinaryClassificationTemplate.ipynb) if you are looking for a template into which you can insert existing models.  OverviewAll machine learning models can be assumed to hold biases, just as all humans hold biases and as all humans fall ill at some point in their lives. The motivation that drives us to study and prevent the harm caused by human illness drives us to prevent the harm caused by innate biases. That means building models that provide fair representation for all demographics. This starts with measurement and evaluation.This notebook introduces concepts, methods, and libraries for measuring fairness in machine learning (ML) models as it relates to problems in healthcare. This is a revamped version of the tutorial presented at the [KDD 2020 Tutorial on Fairness in Machine Learning for Healthcare](../docs/publications/KDD2020-FairnessInHealthcareML-Slides.pptx), the notebook for which can be found here: [/docs/publications/KDD2020-FairnessInHealthcareML-TutorialNotebook.ipynb](../docs/publications/KDD2020-FairnessInHealthcareML-TutorialNotebook.ipynb).[Part 1](part1) will frame the problem and introduce our hypothetical example predicting Length of Stay (LOS) based on a subset of the [MIMIC-III clinical database](https://mimic.physionet.org/gettingstarted/access/). In [Part 2](part2) this baseline model is used as an example to evaluate and discuss common metrics like *Disparate Impact*, *Equalized Odds*, and *Consistency Scores*. Then, [Part 4] compares the results of the baseline model to results for other modeling approaches: one "unaware" version, a "fairness-aware" version, and a third version simply using a different standard machine learning algorithm. There are abundant other publications covering the theoretical basis for fairness metrics, and many resources both online and academic covering the details of specific fairness measures (See [References (bottom)](references) and [Additional Resources (bottom)](additional_resources), or [Our Resources Page](../docs/Measures_QuickReference.md) for just a few). Many of these otherwise excellent references stop short of discussing  edge cases and the practical and philosophical considerations raised when evaluating real models for real customers. Here we attempt to bridge that gap. Table of Contents[Part 1](part1) - Framing the Problem  [Part 2](part2) - Evaluating the Baseline Model[Part 3](part3) - Comparing Models   [References](references)   RequirementsTo run this notebook, please install FairMLHealth using [the instructions posted in GitHub](https://github.com/KenSciResearch/fairMLHealthinstallation_instructions). Some components of this notebook additionally require the [Fairlearn](https://github.com/fairlearn/fairlearn) package.The tutorial uses data from the MIMIC III Critical Care database, a freely accessible source of electronic health records from Beth Israel Deaconess Medical Center in Boston. To download the MIMIC III data, please use this link: [Access to MIMIC III](https://mimic.physionet.org/gettingstarted/access/) and save the data with the default directory name ("MIMIC"). No further action is required beyond remembering the download location, and you do not need to unzip any files.A basic knowledge of ML implementation in Python is assumed.  --------  Part 1 - Framing the Problem<jupyter_code># Standard Libraries
from IPython.display import Markdown, HTML
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
from scipy import stats
import sys
import warnings

# Load Prediction Libraries
import sklearn.metrics as sk_metric
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

# Remove limit to the number of columns and column widths displayed by pandas
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', 0) # adjust column width as needed<jupyter_output><empty_output><jupyter_text>Below are helper functions that make the tutorial easier to read<jupyter_code># Helpers from local folder
from fairmlhealth.mimic_data import load_mimic3_example
from fairmlhealth.model_comparison import measure_model, compare_models
from fairmlhealth import reports
from fairmlhealth.reports import flag 


# Verify that all required packages are present
from fairmlhealth.utils import validate_notebook_requirements, feature_table
validate_notebook_requirements()

# Functions and pointers to make this tutorial more colorful
ks_magenta = '#d00095'
ks_magenta_lt = '#ff05b8'
ks_purple = '#947fed'<jupyter_output><empty_output><jupyter_text>Loading the MIMIC III Data Subset As mentioned above, the MIMIC-III data download contains a folder of zipped_files. The tutorial code in the [Data Subset](datasubset) section below will automatically unzip and format all necessary data for these experiments, saving the formatted data in the same MIMIC folder. Simply enter the correct path of the MIMIC folder in the following cell to enable this feature. Your path should end with the directory "MIMIC".Example: path_to_mimic_data_folder = "~/data/MIMIC"<jupyter_code># path_to_mimic_data_folder = "[path to your downloaded data folder]"
path_to_mimic_data_folder = "~/data/MIMIC"<jupyter_output><empty_output><jupyter_text>Example models in this notebook use data from all years of the MIMIC-III dataset for patients aged 65 and older. Data are imported at the encounter level with all additional patient identification dropped. All models include an "AGE" feature, simplified to 5-year bins, as well as Boolean diagnosis and procedure features categorized through the Clinical Classifications Software system ([HCUP](https://www.hcup-us.ahrq.gov/toolssoftware/ccs/ccs.jsp)). All features other than age are one-hot encoded and prefixed with their variable type (e.g. "GENDER_", "ETHNICITY_").<jupyter_code># Load data and keep a 10K observation subset to speed processing
df = load_mimic3_example(path_to_mimic_data_folder) 
df = df.sample(n=10000, random_state=42).reset_index(drop=True)

# Subset to ages 65+
df = df.loc[df['AGE'].ge(65), :]
df.drop('GENDER_F', axis=1, inplace=True) # Redundant with GENDER_M

# Display insights
display(feature_table(df))
print("\n\n", "Below is a scrollable version of the first five rows of data:")
display(df.head())<jupyter_output>This data subset has 4834 total observations and 648 input features<jupyter_text>Generating the Model<jupyter_code># Display LOS Distributions
ax = df.groupby('LANGUAGE_ENGL')['length_of_stay'
        ].plot(kind='kde', 
               title="Distribution of Length of Stay (in Days) Relative to English Language Status")
plt.show()
# Subset and Split Data
baseline_cols = [c for c in df.columns if (c.startswith('AGE') or c.startswith('DIAGNOSIS_') or c.startswith('PROCEDURE_') or c == "LANGUAGE_ENGL")]
X = df.loc[:, baseline_cols]
y = df['length_of_stay'].clip(0, 25)

#
splits = train_test_split(X, y, test_size=0.33, random_state=42)
X_train, X_test, y_train, y_test = splits
# Train New Model with Language Feature
baseline_model = LinearRegression()
baseline_model.fit(X_train, y_train)
y_pred = baseline_model.predict(X_test)<jupyter_output><empty_output><jupyter_text>FairMLHealth Stratified Data ReporterFairMLHealth includes stratified reporting features to aid in identifying the source of unfairness or other bias. The data reporter evaluates basic statistics specific to each feature-value, in addition to relative statistics for the target value. Since the reporter can evaluate many features at once, it can be a useful option for identifying patterns of bias either alone or in concert with other (e.g., visual methods).<jupyter_code>print("FairMLHealth Stratified Data Report")
reports.data_report(X_test, y_test, features=['LANGUAGE_ENGL'])<jupyter_output>FairMLHealth Stratified Data Report<jupyter_text>-------- Part 2 - Evaluating the Baseline Model In this section we will evaluate an array of fairness measures for a single model. For more information on how to interpret these results, see [Evaluating Fairness](docs/Evaluating_Fairness.md) in our documentation. Skip ahead to [Part 3](part3) for an example comparing multiple models against each other.Our experiment tests sociodemographic bias as it relates to language. Here we suppose that individuals who speak English may be given preferential treatment in an English-speaking society due to the requirement of using a translator. Language may also be used as a proxy for race, religion, or nationality, which are [explicitly protected attributes](https://www.eeoc.gov/employers/small-business/3-who-protected-employment-discrimination).<jupyter_code>lang_values = X_test['LANGUAGE_ENGL']
print("FairMLHealth Fairness Measure Report: English Language")
fairness_report = measure_model(X_test, y_test, lang_values, baseline_model, pred_type="regression")<jupyter_output>FairMLHealth Fairness Measure Report: English Language<jupyter_text>FairMLHealth Stratified Performance ReporterThe stratified performance reporter evaluates model performance specific to each feature-value subset. In this example we can see that the Equalized Odds ratio is out of range because of the significant difference in False Positive Rates (FPR) between the two classes. More about this difference in [Comparing Group Fairness Measures](comparing_group_measures) below.Note that if prediction probabilities (via the *predict_proba()* method) are available to the model, additional ROC_AUC and PR_AUC values will be included in the report.<jupyter_code>print("FairMLHealth Stratified Performance Report")
reports.performance_report(X_test, y_test, y_pred, features=['LANGUAGE_ENGL'], pred_type="regression")<jupyter_output>FairMLHealth Stratified Performance Report<jupyter_text>FairMLHealth Stratified Fairness ReporterThe stratified bias reporter evaluates prediction bias specific to each feature-value subset. It assumes each feature-value as the "privileged" group relative to all other possible values for the feature. To simplify the report, fairness measures have been simplified to their component parts. For example, measures of Equalized Odds can be determined by combining the True Positive Rate (TPR) Ratios & Differences with False Positive Rate (FPR) Ratios & Differences.<jupyter_code>print("FairMLHealth Stratified Fairness Report")
reports.bias_report(X_test, y_test, y_pred, features=['LANGUAGE_ENGL'], pred_type="regression")
quantiles = pd.qcut(y_test, 3, labels=False)

reports.bias_report(X_test, y_test, y_pred, features=['LANGUAGE_ENGL'], 
                    pred_type="regression", cohorts=quantiles)<jupyter_output>/Users/christineallen/repos/fairMLHealth/fairmlhealth/reports.py:941: FutureWarning: this method is deprecated in favour of `Styler.format(precision=..)`
  styled = styled.set_precision(sig_fig)
{'fraction_non_alphanumeric': 0.0679799489930525, 'fraction_numerical': 0.007211327060065078, 'mean_word_length': 6.085358255451713, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '326443', 'license': 'MIT', 'n_tokens_mistral': 3166, 'n_tokens_neox': 2931, 'n_words': 1266, 'path': 'tutorials_and_examples/Tutorial-EvaluatingFairnessInRegression.ipynb', 'repo_name': 'christianliraatkensci/fairMLHealth', 'size': '11371'}