<jupyter_start><jupyter_text>An Introduction to Ethical Supervised Learning ***  OverviewIn this tutorial, you will learn:1. How to identify a discriminatory machine learning predictor2. An introduction to fairness models in supervised learning3. How to fix a basic discriminatory predictor with a fairness model through:    * Understanding perfmorance metrics    * Minimizing loss    * Utilizing sensitivity / recall ***  Part 1 : How to Identify a Discriminatory Predictor  I. Introduction to TransRisk Scores For this tutorial, we will be working with **TransRisk Scores** for four demographics. TransRisk scores aim to estimate how worthy a consumer is for credit. The scores range from 0 to 100, and are created using a variety of data points (such as history of paying back loans, how many credit card accounts you've had, employment rate, etc.) In theory, the closer a person's TransRisk score is to 100, the more likely they are to pay back a loan (similar to a FICO credit score). Banks and loan agencies typically use this kind of score combined with a **threshold value** to decide if someone should be granted a loan.For example, a bank that grants loans might set their threshold TransRisk score at 60. Then everyone with a TransRisk score of 60 or above recieves a loan, and everyone with a score below 60 automatically gets denied.The first data set you will work with shows the probability of an individual having a TransRisk score for different demographic groups. This can also be written as : $$ P\text{(Score} = x \text{ | race)} $$<jupyter_code>import pandas as pd
%matplotlib inline
import numpy as np
# ProbScoreEqualsXGivenRace = pd.read_csv("ProbScoreEqualsXGivenRace.csv").set_index("Score")
ProbScoreEqualsXGivenRace = pd.read_csv("NEW_ProbSEqXGivenR.csv").set_index("Score")
ProbScoreEqualsXGivenRace.head()<jupyter_output><empty_output><jupyter_text>II. Using Visualizations as a Guide *** For a threshold score x, someone is given a loan if they have a TransRisk score >= x. Using this data set, create and plot a dataframe that shows the probability of an individual receiving a loan: ***$$ P\text{(score} \geq x \text{ | race)} $$<jupyter_code>ProbOfGettingLoan = ProbScoreEqualsXGivenRace.iloc[::-1].cumsum()[::-1]
ProbOfGettingLoan.plot.line()<jupyter_output><empty_output><jupyter_text>III. Interpretation Using our previous example of a threshold score x = 60, take a look at your visualization. With a score of 60, what is the probability for an individual of each race to receive a loan?<jupyter_code># Show Probabilities Here
ProbOfGettingLoan.loc[60]<jupyter_output><empty_output><jupyter_text>Keep in mind this isn't an indication if discrimination is occurring or not. If a bank chooses a threshold score of 60, the white demographic would be more than 4x as likely to get a loan than the black demographic. This could happen for a variety of reasons, including (but not limited to):- The data that was used to create a TransRisk score (such as amount of bank accounts opened) might unintentionally discriminate against certain demographic groups.- Machine learning predictors are inherently less accurate for minority populations, since smaller population sizes lead to lower predictor performance.- For a person have a score of 60 or above in the hispanic or black demographic, they likely will need to be in a much higher percentile of their overall population as compared to the white or asian demographic** To determine if a predictor is discriminatory, it must fail the requirements provided by a fairness model. ** ***  Part 2 : Introduction to Fairness Models  I. What is a fairness model? Fairness models seek to fix predictors for discrimination. Specifically, they allow data scientists to algorithmically and statistically measure how *fair* a predictor is by providing requirements to ensure *fairness*. Using these requirements, we can also determine if a predictor is *unfair* (or discriminatory).  II. Equal Opportunity Model This model will work well for the TransRisk dataset, because it provides simple requirements for a binary predictor. This model takes into account three factors:1. Groups that are defined by a protected attribute (this will be the four demographic groups in our example) | Denoted as 'A'2. The advantaged outcome group (we will discuss this in Part 3) | Denoted as 'Y = 1'3. The binary predictor (getting granted a loan) | Denoted as '$\hat Y$ = 1' for a loan approval and '$\hat Y$ = 0' for loan denial A Predictor is said to satisfy fairness under Equal Opportunity iff the probability of getting a positive prediction is the same for all advantaged outcome members in a protected group. $$ P(\hat Y = 1 | A = a, Y = 1) \text{ is equal for ALL protected groups a.} $$ ***  Part 3 : Fixing a Discriminatory Predictor Through the Equal Opportunity Model  I. Understanding Performance Metrics ![Probabilities](ethicalmldiagram.png) Here we will introduce two more data sets on these TransRisk scores. 1. **ProbNonDefaultGivenRace**: P(NonDefault | race)    - Represents the probability that an individual of a race is non-defaulting    - *Note that the non-defaulters represent the advantaged outcome group defined in Part 2, for all non-defaulting individuals of a race,'Y = 1'* 2. **ProbNonDefaultGivenScoreGreaterXAndRace** : P(NonDefault | score >= x, race)    - Represents the probability that an individual who has been granted a loan is actually a non-defaulter    - *Remember that with a threshold score x, everyone with a score >= x is granted a loan.*Read them in and make sure you understand what the data represents before moving onto the next section.<jupyter_code># ProbNonDefaultGivenRace = pd.read_csv("ProbNonDefaultGivenRace.csv")
ProbNonDefaultGivenRace = pd.read_csv("NEW_pis.csv")

# ProbLoanReceiverIsNonDefault = (
#     pd.read_csv("ProbNonDefaultGivenScoreGreaterXAndRace.csv")
#     .set_index('TransRisk Score'))

ProbLoanReceiverIsNonDefault = (
    pd.read_csv("NEW_ProbNDGivenSGEqXAndR.csv")
    .set_index('Score'))
ProbNonDefaultGivenRace
ProbLoanReceiverIsNonDefault.head()
ProbOfGettingLoan.head()<jupyter_output><empty_output><jupyter_text>***   The General Multiplication Rule:  **Definition : ** For any two events, we can say that$$ P(A \text{ and } B) = P(A) * P(B | A)$$*"The probability that both A and B occur = the probability that A occurs multiplied by the probability that B occurs, given that A has occured."* *** To work with the conditions introduced by the **Equal Opportunity Model**, we need to find$$ P(\hat Y = 1 | A = a, Y = 1)$$For our case study, this would be the Probability that someone has received a loan, given their race and the fact that they are non-defaulting. Also stated:$$ P(\text{score} \geq x | \text{ race, NonDefault)}$$ *If you look at the visual table of probabilities above, this is shown with the **PURPLE** boxes.*Using your knowledge of the multiplication rule with all three data sets you have been given on TransRisk scores, create a DataFrame that shows this probability for each TransRisk score and demographic, then plot it to see what this data looks like.<jupyter_code>ProbOfGettingLoan.columns = ProbOfGettingLoan.columns.str.lower()
ProbLoanReceiverIsNonDefault.columns = ProbLoanReceiverIsNonDefault.columns.str.lower()
demographics = ['white', 'black', 'asian', 'hispanic']
s = []
for demographic in demographics:
        s.append(ProbOfGettingLoan[[demographic]] * ProbLoanReceiverIsNonDefault[[demographic]] 
        / ProbNonDefaultGivenRace[demographic][0])
        
# for demographic in demographics:
#         s.append(ProbOfGettingLoan[[demographic]] * ProbLoanReceiverIsNonDefault[[demographic]]) 
        
# sensitivites rounded to two decimal points so they can be more easily used as indices
sensitivities = pd.concat(s, axis= 1).round(2)
sensitivities.head()
sensitivities.plot.line()<jupyter_output><empty_output><jupyter_text>** Now, restructure this dataframe so the sensitivity values are the index, and each demographic has one TransRisk Score (threshold value) associated with each sensitivity **<jupyter_code># creating a dataframe with all possible rounded sensitivity values to be the index
totalDF = pd.DataFrame({
    'sensitivity': np.arange(0.0, 1.01, 0.01).tolist()
})
# function to format the threshold scores to match the sensitivity value at the index
def getScoreArray(race, data):
    scores = [None] * 101
    for index, row in data.iterrows():
        sens = data.loc[index][race]
        if(len(totalDF[totalDF['sensitivity'] == sens].index.values) >= 1):
            i = totalDF[totalDF['sensitivity'] == sens].index.values[0]
            scores[i] = index
    return scores
for demographic in demographics:
    totalDF['t_' + demographic]  = getScoreArray(demographic, sensitivities[[demographic]])
totalDF.set_index('sensitivity').head()<jupyter_output><empty_output><jupyter_text>II. Minimizing Loss For supervised learning predictors, one way to detect errors is to calculate the difference between the actual output and the predictor's output. The function used to compute the error of a predictor is called a **Loss Function**. For our dataset, we calculate loss in a similar manner to how banks giving loans would. Loss is calculated using False Positives (FP) and False Negatives (FN), which means every loss value can be associated with a precision value. Generally, banks strive for a maximum loss of 0.18, which correlates to a precision of 0.82. So, we want to make the ** *Precision = 0.82* **.$$ \text{Precision   ==   P(Y} = 1 | \hat Y = 1) \text{  ==   P(NonDefault | score} \geq x) $$Here we provide another dataset that shows the percentage of each demographic in the total sample size, or rather P(race). Using this information and the multiplication rule, find the precision for every demographic at every sensitivity value. So you should create four new columns in your sensitivity dataframe from before, one for each demographic precision.<jupyter_code># This represents P(race)
# ProbOfBeingRace = pd.read_csv("ProbOfBeingRace.csv").set_index("Demographic")
ProbOfBeingRace = pd.read_csv("NEW_ProbR.csv").set_index("Demographic")
ProbOfBeingRace
# function to order the precisions for each race in association with the demographic specific threshold value at
# each index. These are then summed for a total precision value to find the final sensitivity
def getPrecisionOrder(t_race, dataPrecision):
    prec = []
    for i in range(len(totalDF)):
        if(totalDF[t_race][i] != None):
            currPrec = dataPrecision[totalDF[t_race][i]]
            if(isinstance(currPrec, np.ndarray)):
                currPrec = 0
            prec.append(currPrec)
    return prec
# Calculate precision
for demographic in demographics:
    precision = ProbLoanReceiverIsNonDefault[demographic] * ProbOfBeingRace.loc[demographic]["P(race)"]
    totalDF[demographic[0] + '_prec'] = getPrecisionOrder('t_' + demographic, precision)
    
totalDF.set_index('sensitivity', inplace=True)
totalDF.head()<jupyter_output><empty_output><jupyter_text>III. Utilizing Sensitivity We finally have all of the peices to fix the discriminatory predictor using the Equal Opportunity Model. As you just learned in Section II, our desired loss directly leads to a precision of 0.82. Looking back at the requirements for Equal Opportunity, we can see it requires that $ P(\hat Y = 1 | A = a, Y = 1)$ is equal for ALL protected groups a. This can also be stated that the **true positive rate** is equal for ALL protected groups a. Or even further, that the ** *Sensitivity* ** is equal for ALL protected groups a.If you look back at your plot of sensitivity values, you can see that choosing one sensitivity value (drawing a horizontal line on your plot) would yeild four threshold values. But where do we draw this line? Which singular sensitivity value do we want?As we saw in the last section, we choose the sensitivity value that gives us a total precision of 0.82. Now you simply need to find the threshold scores associated with that sensitivity value for each demographic. These scores represent the threshold that needs to be set for that particular demographic in order to satisfy 'fairness'. Finish by recording the final four threshold values needed to satisfy the Equal Opportunity Model. ** First: Aggregate the four precisions at each sensitivity value for a TOTAL PRECISION column in your dataframe. **<jupyter_code>totalDF['total_prec'] = totalDF[['w_prec', 'b_prec', 'a_prec', 'h_prec']].sum(axis=1)
totalDF.head()<jupyter_output><empty_output><jupyter_text>*** What four sensitivity values are associated with a precision of 0.82? *** (put answer below)<jupyter_code>closestPrecision = min(totalDF["total_prec"], key=lambda x:abs(x-0.82))
totalDF[totalDF['total_prec'] == closestPrecision]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0757906215921483, 'fraction_numerical': 0.007088331515812432, 'mean_word_length': 4.971627906976744, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '349757', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3602, 'n_tokens_neox': 3271, 'n_words': 1597, 'path': '.ipynb_checkpoints/Tutorial-Solution-Revised-checkpoint.ipynb', 'repo_name': 'jesmith14/fairness-ml-tutorial', 'size': '12838'}