<jupyter_start><jupyter_text>Fair Dummies: Regression ExampleThis notebook implements a the fair dummies framework for learning predictive models that approximately satisfy the equalized odds notion of fairness.Paper: "Achieving Equalized Odds by Resampling Sensitive Attributes," <NAME>, <NAME>, and <NAME>, 2020 Proposed approach__Core idea__: fit a regression function, minimizing$$ \text{loss = prediction error + distance to equalized odds}$$__Input__: $  \{(X_i,A_i,Y_i)\}_{i=1}^n \sim P_{XAY}$ training data    __Step 1__: sample dummy protected attributes$$\tilde{A}_i \sim P_{A|Y}(A \mid Y=y_{i}) \quad \forall \ i=1,2,\dots,n \nonumber$$$A \in \{0,1\}$? generate $\tilde{A}$ using a biased coin-flip, with$$P\{A=1|Y=y\} = \frac{P\{y \mid A=1\}P\{A=1\}}{P\{y \mid A=1\}P\{A=1\} + P\{y \mid A=0\}P\{A=0\}}$$    __Step 2__: fit a regression function on $\{(X_i, A_i, Y_i)\}_{i=1}^n$$$        \hat{f}(\cdot) \,= \, \underset{f \in \mathcal{F}}{\mathrm{arg min}} \, \frac{1-\lambda}{n} \sum_{i=1}^n (Y_i - f(X_i))^2  + \lambda \mathcal{D}\left( [\hat{\mathbf{Y}}, \mathbf{A}, \mathbf{Y}] , [\hat{\mathbf{Y}}, \tilde{\mathbf{A}}, \mathbf{Y}] \right) \nonumber$$where    $$\hat{\mathbf{Y}} = \left[f(X_{1}), f(X_{2}), \dots, f(X_{n})\right]^T \ ; \ \mathbf{A} = \left[A_{1}, A_{2}, \dots, A_{n}\right]^T \ ; \ \tilde{\mathbf{A}} = \left[\tilde{A}_{1}, \tilde{A}_{2}, \dots, \tilde{A}_{n}\right]^T \ ; \ \mathbf{Y} = \left[Y_{1}, Y_{2}, \dots, Y_{n}\right]^T$$and $\mathcal{D}\left( \mathbf{Z}_1, \mathbf{Z}_2 \right)$ tests whether $P_{Z_1} = \ P_{Z_2}$ given $\mathbf{Z}_1, \mathbf{Z}_2$,  here it implemented as a classifier two-sample test + second moment penalty (see paper for more details)<jupyter_code>import os
import sys

base_path = os.getcwd() + '/data/'

import torch
import random
import get_dataset
import numpy as np
import pandas as pd
from fair_dummies import utility_functions
from fair_dummies import fair_dummies_learning

seed = 123
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(seed)

dataset = "crimes"

# use all data in sgd (without minibatches)
batch_size = 10000

# step size to minimize loss
lr_loss = 0.01

# step size used to fit binary classifier (discriminator)
lr_dis = 0.01

# inner epochs to fit loss
loss_steps = 80

# inner epochs to fit discriminator classifier
dis_steps = 80

# equalized odds penalty
mu_val = 0.7
second_scale = 1


# total number of epochs
epochs = 20

# utility loss
cost_pred = torch.nn.MSELoss()

# base predictive model
model_type = "linear_model"<jupyter_output><empty_output><jupyter_text>Load data<jupyter_code>print("dataset: " + dataset)

X, A, Y, X_cal, A_cal, Y_cal, X_test, A_test, Y_test = get_dataset.get_train_test_data(base_path, dataset, seed)

in_shape = X.shape[1]

print("training samples (used to fit predictive model) = " + str(X.shape[0]) + " p = " + str(X.shape[1]))
print("holdout samples (used to fit fair-dummies test statistics function) = " + str(X_cal.shape[0]))
print("test samples = " + str(X_test.shape[0]))
model = fair_dummies_learning.EquiRegLearner(lr=lr_loss,
                                             pretrain_pred_epochs=0,
                                             pretrain_dis_epochs=0,
                                             epochs=epochs,
                                             loss_steps=loss_steps,
                                             dis_steps=dis_steps,
                                             cost_pred=cost_pred,
                                             in_shape=in_shape,
                                             batch_size=batch_size,
                                             model_type=model_type,
                                             lambda_vec=mu_val,
                                             second_moment_scaling=second_scale,
                                             out_shape=1)

input_data_train = np.concatenate((A[:,np.newaxis],X),1)
model.fit(input_data_train, Y)
input_data_cal = np.concatenate((A_cal[:,np.newaxis],X_cal),1)
Yhat_out_cal = model.predict(input_data_cal)

input_data_test = np.concatenate((A_test[:,np.newaxis],X_test),1)
Yhat_out_test = model.predict(input_data_test)
rmse_trivial = np.sqrt(np.mean((np.mean(Y_test)-Y_test)**2))
print("RMSE trivial = " + str(rmse_trivial))

rmse = np.sqrt(np.mean((Yhat_out_test-Y_test)**2))
print("RMSE trained model = " + str(rmse))

p_val = utility_functions.fair_dummies_test_regression(Yhat_out_cal,
                                                       A_cal,
                                                       Y_cal,
                                                       Yhat_out_test,
                                                       A_test,
                                                       Y_test,
                                                       num_reps = 1,
                                                       num_p_val_rep=1000,
                                                       reg_func_name="Net")<jupyter_output>RMSE trivial = 0.23858692405157056
RMSE trained model = 0.15518249498501968
Init Loss = 0.7545293
Final Loss = 0.011895132
Fair dummies test (regression score), p-value: 0.25774225774225773
{'fraction_non_alphanumeric': 0.1330039901197036, 'fraction_numerical': 0.028310849325479764, 'mean_word_length': 2.2254901960784315, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 40, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '94772', 'license': 'MIT', 'n_tokens_mistral': 1752, 'n_tokens_neox': 1646, 'n_words': 370, 'path': 'real_regression_experiment.ipynb', 'repo_name': 'altosaar/fair_dummies', 'size': '5274'}