<jupyter_start><jupyter_text>Explaining Measures of Fairness with SHAPThis hands-on article connects explainable AI methods with fairness measures and shows how modern explainability methods can enhance the usefulness of quantitative fairness metrics. By using [SHAP](http://github.com/slundberg/shap) (a popular explainable AI tool) we can decompose measures of fairness and allocate responsibility for any observed disparity among each of the model's input features. Explaining these quantitative fairness metrics can reduce the concerning tendency to rely on them as opaque standards of fairness, and instead promote their informed use as tools for understanding how model behavior differs between groups.Quantitative fairness metrics seek to bring mathematical precision to the definition of fairness in machine learning [[1](https://books.google.com/books/about/The_Ethical_Algorithm.html?id=QmmtDwAAQBAJ&source=kp_book_description)]. Definitions of fairness however are deeply rooted in human ethical principles, and so on value judgements that often depend critically on the context in which a machine learning model is being used. This practical dependence on value judgements manifests itself in the mathematics of quantitative fairness measures as a set of trade-offs between sometimes mutually incompatible definitions of fairness [[2](https://arxiv.org/abs/1609.05807)]. Since fairness relies on context-dependent value judgements it is dangerous to treat quantitative fairness metrics as opaque black-box measures of fairness [[3](https://arxiv.org/abs/1808.00023)], since doing so may obscure important value judgment choices.<!--This article covers:1. How SHAP can be used to explain various measures of model fairness.2. What SHAP fairness explanations look like in various simulated scenarios.3. How introducing a protected feature can help distiguish between label bias vs. feature bias. 4. Things you can't learn from a SHAP fairness explanation.--> How SHAP can be used to explain various measures of model fairnessThis article is not about how to choose the correct measure of model fairness, but rather about explaining whatever metrics you have chosen. Which fairness metric is most appropriate depends the specifics of your context, such as what laws apply, how the outputs of the machine learning model impact people, and what value you place on various outcomes and hence tradeoffs. For this analysis we use the classic [demographic parity](https://arxiv.org/pdf/1803.02453.pdf) metric, though the same analysis works with other metrics such as [equalized odds](https://arxiv.org/pdf/1803.02453.pdf), [equal opportunity](https://ttic.uchicago.edu/~nati/Publications/HardtPriceSrebro2016.pdf), or [equal quality of service](https://github.com/fairlearn/fairlearn/blob/master/TERMINOLOGY.md). Demographic parity states that the output of the machine learning model should be equal between two or more groups. The demographic parity difference is then a measure of how much disparity there is between model outcomes in two groups of samples. We use demographic parity for this scenario not because it is the right measure of fairness in most situations (it is often not), but because it is straightforward and commonly known. In our simulated case demographic parity is appropriate because in our simulation the true labels (will default on a loan) are statistically independent of sex (the sensitive class we use to check for fairness).**Since SHAP decomposes the model output into feature attributions with the same units as the original model output, we can first decompose the model output among each of the input features using SHAP, and then compute the demographic parity difference (or any other fairness metric) for each input feature seperately using the SHAP value for that feature.** Because the SHAP values sum up to the model's output, the sum of the demographic parity differences of the SHAP values also sum up to the demographic parity difference of the whole model.<!--To  will not explainThe danger of treating quantitative fairness metrics as opaque, black-box measures of fairness is strikingly similar to a related problem of treating machine learning models themselves as opaque, black-box predictors. While using a black-box is reasonable in many cases, important problems and assumptions can often be hidden (and hence ignored) when users don't understand the reasons behind a model's behavior \cite{ribeiro2016should}. In response to this problem many explainable AI methods have been developed to help users understand the behavior of modern complex models \cite{vstrumbelj2014explaining,ribeiro2016should,lundberg2017unified}. Here we explore how to apply explainable AI methods to quantitative fairness metrics.--> What SHAP fairness explanations look like in various simulated scenariosTo help us explore the potential usefulness of explaining quantitative fairness metrics we consider a simple simulated scenario based on credit underwriting. In our simulation there are four underlying factors that drive the risk of default for a loan: income stability, income amount, spending restraint, and consistency. These underlying factors are not observed, but they variously influence four different observable features: job history, reported income, credit inquiries, and late payments. Using this simulation we generate random samples and then train a non-linear [XGBoost](https://xgboost.ai/) classifier to predict the probability of default. The same process also works for any other model type supported by SHAP, just remember that explanations of more complicated models hide more of the model's details.By introducing sex-specific reporting errors into a fully specified simulation we can observe how the biases caused by these errors are captured by fairness metrics. In our simulated case we can appropriatly use the demographic parity metric because in our simulation the true labels (will default on a loan) are statistically independent of sex (the sensitive class we use to check for fairness). If the true labels you are predicting (which might be different than the training labels you have access to) are not (marginally) statistically independent of the sensitive feature you are considering, then even a perfect model with no errors would fail demographic parity, and so you will likely want to consider other measures of fairness.<!--This article explores how we can use modern explainable AI tools to enhance traditional quantitative measures of model fairness. It is practical and hands-on, so feel free to follow along in the associated [notebook]. I assume you have a basic understanding of how people measure fairness for machine learning models. If you have never before considered fairness in the context of machine learning, then I recommend starting with a basic introduction such as XXX. I am not writing this Here I do not  beforeIt is not meant to be a definitite  One futher disclaimer is that as the author of SHAP (a popular explainable AI tool) I am very familar with the strengths and weaknesses of explainable AI tools, but I do not consider myself a fairness expert. So consider this a thought-provoking guide on how explainable AI tools can enhance quantitative measures of model fairnessI consider myself fairly well informed about explainable AI, but I Questions about fairness and equal treatment naturally arise whenever the outputs of a machine learning model impact people. For sensitive use-cases such as credit underwriting or crime prediction there are even laws that govern certain aspects of fairness. While fairness issues are not new, the rising popularily of machine learning model  Legal fairness protections are even legally encorced for sensitive use-cases such as credit underwriting or crime prediction, but is also important in many other situations such as quality of service, or you might not initially to consider whenever you are using m Quantifying the fairness of a machine learning model has recently received considerable attention in the research community, and many quantitative fairness metrics have been proposed. In parallel to this work on fairness, explaining the outputs of a machine learning model has also received considerable research attention. %Explainability is intricately connected to fairness, since good explanations enable users to understand a model's behavior and so judge its fairness.Here we connect explainability methods with fairness measures and show how recent explainability methods can enhance the usefulness of quantitative fairness metrics by decomposing them among the model's input features. Explaining quantitative fairness metrics can reduce our tendency to rely on them as opaque standards of fairness, and instead promote their informed use as tools for understanding model behavior between groups.  This notebook explores how SHAP can be used to explain quantitative measures of fairness, and so enhance their usefulness. To do this we consider a simple simulated scenario based on credit underwriting. In the simulation below there are four underlying factors that drive the risk of default for a loan: income stability, income amount, spending restraint, and consistency. These underlying factors are not observed, but they influence four different observable features in various ways: job history, reported income, credit inquiries, and late payments. Using this simulation we generate random samples and then train a non-linear gradient boosting tree classifier to predict the probability of default.By introducing sex-specific reporting errors into the simulation we can observe how the biases caused by these errors are captured by fairness metrics. For this analysis we use the classic statistical parity metric, though the same analysis works with other metrics. Note that for a more detailed description of fairness metrics you can check out the [fairlearn package's documentation](https://github.com/fairlearn/fairlearn/blob/master/TERMINOLOGY.mdfairness-of-ai-systems).--><jupyter_code># here we define a function that we can call to execute our simulation under
# a variety of different alternative scenarios
import scipy as sp
import numpy as np
import matplotlib.pyplot as pl
import pandas as pd
import shap
%config InlineBackend.figure_format = 'retina'
def run_credit_experiment(N, job_history_sex_impact=0, reported_income_sex_impact=0, income_sex_impact=0,
                          late_payments_sex_impact=0, default_rate_sex_impact=0,
                          include_brandx_purchase_score=False, include_sex=False):
    np.random.seed(0)
    sex = np.random.randint(0, 2, N) == 1 # randomly half men and half women
    
    # four hypothetical causal factors influence customer quality
    # they are all scaled to the same units between 0-1
    income_stability = np.random.rand(N)
    income_amount = np.random.rand(N)
    if income_sex_impact > 0:
        income_amount -= income_sex_impact/90000 * sex * np.random.rand(N)
        income_amount -= income_amount.min()
        income_amount /= income_amount.max()
    spending_restraint = np.random.rand(N)
    consistency = np.random.rand(N)
    
    # intuitively this product says that high customer quality comes from simultaneously
    # being strong in all factors 
    customer_quality = income_stability * income_amount * spending_restraint * consistency
    default_rate = -customer_quality 
    
    # job history is a random function of the underlying income stability feature
    job_history = np.maximum(
        10 * income_stability + 2 * np.random.rand(N) - job_history_sex_impact * sex * np.random.rand(N)
    , 0)
    
    # reported income is a random function of the underlying income amount feature
    reported_income =  np.maximum(
        10000 + 90000*income_amount + np.random.randn(N) * 10000 - \
        reported_income_sex_impact * sex * np.random.rand(N)
    , 0)
    
    # credit inquiries is a random function of the underlying spending restraint and income amount features
    credit_inquiries = np.round(6 * np.maximum(-spending_restraint + income_amount, 0)) + \
                       np.round(np.random.rand(N) > 0.1)
    
    # credit inquiries is a random function of the underlying consistency and income stability features
    late_payments = np.maximum(
        np.round(3 * np.maximum((1-consistency) + 0.2 * (1-income_stability), 0)) + \
        np.round(np.random.rand(N) > 0.1) - np.round(late_payments_sex_impact * sex * np.random.rand(N))
    , 0)
    
    # bundle everything into a data frame and define the labels based on the default rate and customer quality
    X = pd.DataFrame({
        "Job history": job_history,
        "Reported income": reported_income,
        "Credit inquiries": credit_inquiries,
        "Late payments": late_payments
    })
    default_rate = 0.40 + sex * default_rate_sex_impact
    y = customer_quality < np.percentile(customer_quality, default_rate * 100) 
    
    if include_brandx_purchase_score:
        brandx_purchase_score = sex + 0.8 * np.random.randn(N)
        X["Brand X purchase score"] = brandx_purchase_score
    
    if include_sex:
        X["Sex"] = sex + 0
    
    # build model
    import xgboost
    model = xgboost.XGBClassifier(max_depth=1, n_estimators=500, subsample=0.5, learning_rate=0.05)
    model.fit(X, y)

    # build explanation
    import shap
    explainer = shap.TreeExplainer(model, shap.sample(X, 100))
    shap_values = explainer.shap_values(X)
    
    return shap_values, sex, X, explainer.expected_value<jupyter_output><empty_output><jupyter_text><!-- Scenario A: No reporting errorsAs a baseline experiment we refrain from introducing any sex-specific reporting errors. This results in no significant statistical parity difference between the credit score of men and women:--> Scenario A: No reporting errorsOur first experiment is a simple baseline check where we refrain from introducing any sex-specific reporting errors. While we could use any model output to measure demographic parity, we use the continuous log-odds score from a binary XGBoost classifier. As expected, this baseline experiment results in no significant demographic parity difference between the credit scores of men and women. We can see this by plotting the difference between the average credit score for women and men as a bar plot and noting that zero is close to the margin of error (note that negative values mean women have a lower average predicted risk than men, and positive values mean that women have a higher average predicted risk than men):<jupyter_code>N = 10000
shap_values_A, sex_A, X_A, ev_A = run_credit_experiment(N)
model_outputs_A = ev_A + shap_values_A.sum(1)
glabel = "Demographic parity difference\nof model output for women vs. men"
xmin = -0.8
xmax = 0.8
shap.group_difference_plot(shap_values_A.sum(1), sex_A, xmin=xmin, xmax=xmax, xlabel=glabel)<jupyter_output><empty_output><jupyter_text>Now we can use SHAP to decompose the model output among each of the model's input features and then compute the demographic parity difference on the component attributed to each feature. As noted above, because the SHAP values sum up to the model's output, the sum of the demographic parity differences of the SHAP values for each feature sum up to the demographic parity difference of the whole model. This means that the sum of the bars below equals the difference between the two bars above (the demographic parity difference of our baseline scenario model).<jupyter_code>slabel = "Demographic parity difference\nof SHAP values for women vs. men"
shap.group_difference_plot(shap_values_A, sex_A, X_A.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output><jupyter_text>Scenario B: An under-reporting bias for women's incomeIn our baseline scenario we designed a simulation where sex had no impact on any of the features or labels used by the model. Here in scenario B we introduce an under-reporting bias for women's income into the simulation. The point here is not how realistic it would be for women's income to be under-reported in the real-world, but rather how we can identify that a sex-specific bias has been introduced and understand where it came from. By plotting the average model output (credit score) for both men and women we can see that the income under-reporting bias has created a significant demographic parity difference:<jupyter_code>shap_values_B, sex_B, X_B, ev_B = run_credit_experiment(N, reported_income_sex_impact=30000)
model_outputs_B = ev_B + shap_values_B.sum(1)
shap.group_difference_plot(shap_values_B.sum(1), sex_B, xmin=xmin, xmax=xmax, xlabel=glabel)<jupyter_output><empty_output><jupyter_text>If this were a real application, this demographic parity difference might trigger an in-depth analysis of the model to determine what might be causing the disparity. While this investigation is challenging given just a single demographic parity difference value, it is much easier given the per-feature statistical parity decomposition based on SHAP. Using SHAP we can see there is a significant bias coming from the reported income feature that is increasing the risk of women disproportionately to men. Using the SHAP explanation of the demographic parity difference we can quickly identify which feature has the reporting bias that is causing our model to violate demographic parity:<jupyter_code>shap.group_difference_plot(shap_values_B, sex_B, X_B.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output><jupyter_text>It is important to note at this point how our assmptions can impact the interpretation of SHAP fairness explanations. In our simulated scenario we know that women actually have identical income profiles to men, so when we see that the reported income feature is biased lower for women than for men, we know that has come from a bias in the measurement errors in the reported income feature. The best way to address this problem would be figure out how to debias the measurement errors in the reported income feature. Doing so would create a more accurate model that also has less demographic disparity. However, if we instead assume that women actually are making less money than men (and it is not just a reporting error), then we can't just "fix" the reported income feature. Instead we have to carefully consider how best to account for real differences in default risk between two protected groups. It is impossible to determine which of these two situations is happening using just the SHAP the fairness explanation, since in both cases the reported income feature will be reponsible for an observed disparity between risks for men and women.<jupyter_code>shap_values_B2, sex_B2, X_B2, ev_B2 = run_credit_experiment(N, income_sex_impact=30000)
model_outputs_B2 = ev_B2 + shap_values_B2.sum(1)
shap.group_difference_plot(shap_values_B2.sum(1), sex_B2, xmin=xmin, xmax=xmax, xlabel=glabel)
shap.group_difference_plot(shap_values_B2, sex_B2, X_B2.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output><jupyter_text>Scenario C: An under-reporting bias for women's late paymentsTo verify that SHAP demographic parity explanations can correctly detect disparies regardless of the direction of effect or source feature, we repeat our previous experiment but instead of an under-reporting bias for income, we introduce an under-reporting bias for women's late payment rates. This results in a significant demographic parity difference for the model's output:<jupyter_code>shap_values_C, sex_C, X_C, ev_C = run_credit_experiment(N, late_payments_sex_impact=2)
model_outputs_C = ev_C + shap_values_C.sum(1)
shap.group_difference_plot(shap_values_C.sum(1), sex_C, xmin=xmin, xmax=xmax, xlabel=glabel)<jupyter_output><empty_output><jupyter_text>And as we would hope, the SHAP explantions correctly highlight the late payments feature as the cause of the model's demographic parity difference, as well as the direction of the effect (in this case under-reporting late payments lowers women's predicted risk of default compared to men):<jupyter_code>shap.group_difference_plot(shap_values_C, sex_C, X_C.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output><jupyter_text>Scenario D: An under-reporting bias for women's default ratesThe experiments above focused on introducing reporting errors for specific input features. Next we consider what happens when introduce reporting errors on the training labels through an under-reporting bias on women's default rates (meaning real defaults are less likely to be reported for women than men). Interestingly, for our simulated scenario this results in no significant demographic parity differences in model's output:<jupyter_code>shap_values_D, sex_D, X_D, ev_D = run_credit_experiment(N, default_rate_sex_impact=-0.1) # 20% change
model_outputs_D = ev_D + shap_values_D.sum(1)
shap.group_difference_plot(shap_values_D.sum(1), sex_D, xmin=xmin, xmax=xmax, xlabel=glabel)<jupyter_output><empty_output><jupyter_text>We also see no evidence of any demographic parity differences in the SHAP explanations.<jupyter_code>shap.group_difference_plot(shap_values_D, sex_D, X_D.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output><jupyter_text>Scenario E: An under-reporting bias for women's default rates, take 2It may at first be surprising that no statistical parity differences were caused when we introduced an under-reporting bias on women's default rates. This is beacause none of the four features in our simulation are significantly correlated with sex, so none of them could be effectively used to model the bias we introduced into the training labels. If we now instead provide a new feature (brand X purchase score) to the model that is correlated with sex, then we see a statistical parity difference emerge as that feature is used by the model to capture the sex-specific bias in the training labels:<jupyter_code>shap_values_E, sex_E, X_E, ev_E = run_credit_experiment(
    N, default_rate_sex_impact=-0.1, include_brandx_purchase_score=True
)
model_outputs_E = ev_E + shap_values_E.sum(1)
shap.group_difference_plot(shap_values_E.sum(1), sex_E, xmin=xmin, xmax=xmax, xlabel=glabel)<jupyter_output><empty_output><jupyter_text>When we explain the statistical parity difference with SHAP we see that, as expected, the brand X purchase score feature drives the difference. In this case it is not because we have a bias in how we measure the brand X purchase score feature, but rather because we have a bias in our training label that gets captured by any input features that are suffciently correlated with sex:<jupyter_code>shap.group_difference_plot(shap_values_E, sex_E, X_E.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output><jupyter_text>Scenario F: Teasing apart multiple under-reporting biasesWhen there is a single cause of reporting bias then both the classic demographic parity test on the model's output, and the SHAP explanation of the demographic parity test capture the same bias effect (though the SHAP explanation can often have more statistical significance since it isolates the feature causing the bias). But what happens when there are multiple causes of bias occuring in a dataset? In this experiment we introduce two such biases, an under-reporting of women's default rates, and an under-reporting of women's job history. These biases tend to offset each other in the global average and so the classic statistical parity test on the model's output shows no measurable disparity:<jupyter_code>shap_values_F, sex_F, X_F, ev_F = run_credit_experiment(
    N, default_rate_sex_impact=-0.1, include_brandx_purchase_score=True,
    job_history_sex_impact=2
)
model_outputs_F = ev_F + shap_values_F.sum(1)
shap.group_difference_plot(shap_values_F.sum(1), sex_F, xmin=xmin, xmax=xmax, xlabel=glabel)<jupyter_output><empty_output><jupyter_text>However, if we look at the SHAP explaination of the statistical parity difference we clearly see both (counteracting) biases:<jupyter_code>shap.group_difference_plot(shap_values_F, sex_F, X_F.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output><jupyter_text>Identifying multiple potentially offsetting bias effects can be important since while on average there is no disparate impact on men or women, there is disparate impact on individuals. For example, in this simulation women who have not shopped at brand X will recive a lower credit score than they should have because of the bias present in job history reporting.  How introducing a protected feature can help distiguish between label bias and feature biasIn scenario F we were able to pick apart two distict forms of bias, one coming from job history under-reporting and one coming from default rate under-reporting. However, the bias from default rate under-reporting was not attributed to the default rate label, but rather to the brand X purchase score feature that happened to be correlated with sex. This still leaves us with some uncertainty about the true sources of statistical parity differences, since any difference attributed to an input feature could be due to an issue with that feature, or due to an issue with the training labels.It turns out that in this case we can help disentangle label bias from feature bias by introducing sex as a variable directly into the model. The goal of introducing sex as an input feature is to cause the label bias to fall entirely on the sex feature, leaving the feature biases untouched. So we can then distinguish between label biases and feature biases by comparing the results of scenario F above to our new scenario G. This of course creates an even stronger demographic parity difference than we had before, but that is fine since our goal here is not bias mitigation, but rather bias understanding.<jupyter_code>shap_values_G, sex_G, X_G, ev_G = run_credit_experiment(
    N, default_rate_sex_impact=-0.1, include_brandx_purchase_score=True,
    job_history_sex_impact=2, include_sex=True
)
model_outputs_G = ev_G + shap_values_G.sum(1)
shap.group_difference_plot(shap_values_G.sum(1), sex_G, xmin=xmin, xmax=xmax, xlabel=glabel)<jupyter_output>99%|===================| 9941/10000 [00:11<00:00]<jupyter_text>The SHAP explanation for scenario G shows that all of the demographic parity difference that used to be attached to the brand X purchase score feature in scenario F has now moved to the sex feature, while none of the demographic parity difference attached to the job history feature in scenario F has moved. This can be interpreted to mean that all of the disparity attributed to brand X purchase score in scenario F was due to label bias, while all of the disparity attributed to job history in scenario F was due to feature bias.<jupyter_code>shap.group_difference_plot(shap_values_G, sex_G, X_G.columns, xmin=xmin, xmax=xmax, xlabel=slabel)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.04756251391259182, 'fraction_numerical': 0.007939452400385842, 'mean_word_length': 5.195127556883475, 'pattern_counts': {'":': 4, '<': 70, '<?xml version=': 0, '>': 71, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '215160', 'license': 'MIT', 'n_tokens_mistral': 7069, 'n_tokens_neox': 6552, 'n_words': 3437, 'path': 'notebooks/general/Explaining Quantitative Measures of Fairness.ipynb', 'repo_name': 'dylan-slack/shap', 'size': '26954'}