<jupyter_start><jupyter_text>This file shows steps to construct synthetic data using the method shown in the paper. We first import packages and set a seed<jupyter_code>import sys
sys.path.append("models")
sys.path.append("AIF360/")
import numpy as np
from scipy.stats import bernoulli
from aif360.datasets import StandardDataset
from fairness_metrics.tot_metrics import TPR, TNR
from aif360.algorithms.preprocessing.optim_preproc import OptimPreproc
from sklearn.linear_model import LogisticRegression
from aif360.metrics import BinaryLabelDatasetMetric
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score
import pandas as pd
from aif360.algorithms.preprocessing.optim_preproc_helpers.opt_tools import OptTools
import warnings
warnings.filterwarnings("ignore")
warnings.simplefilter("ignore", UserWarning)
from compas_model import get_evaluation
np.random.seed(10)<jupyter_output><empty_output><jupyter_text>We then define label column. In this example, I have 10000 0's and 10000 1's for the label column. 0 is the negative outcome and 1 is the positive outcome. We also need to define the sensitive attribute value and the correlation between sensitive attribute and the outcome<jupyter_code>Y = [0] * 10000
Y.extend([1] * 10000)
# S here is the sensitive attribute where 0 is the unprivileged group and 1 is the privileged group
S = []
for i in Y:
    if i == 0:
        S.append(bernoulli.rvs(0.35))
    else:
        S.append(bernoulli.rvs(0.6))<jupyter_output><empty_output><jupyter_text>The next section is to define non-sensitive features for prediction. In non-sensitive features, we need to determine the correlation between the non-sensitive feature and outcome, sensitive attibute and other non-sensitive features<jupyter_code>df = pd.DataFrame()
df['y'] = Y
df['sens'] = S
df['tmp'] = 1
df.groupby(['y', 'sens']).count()
tmp = []
for i, j in zip(df['y'], df['sens']):
    if i == 0 and j == 0:
        tmp.append(np.random.poisson(lam=6))
    if i == 1 and j == 0:
        tmp.append(np.random.poisson(lam=3))
    if i == 0 and j == 1:
        tmp.append(np.random.poisson(lam=4))
    if i == 1 and j == 1:
        tmp.append(np.random.poisson(lam=2))
df['edu_orig'] = tmp
tmp = df['edu_orig'].sort_values().to_list()

# create 4 equal-width bins to bin continuous data into categorical
def cat_edu(x):
    if x <= tmp[int(len(tmp) / 4)]:
        return 'edu_cat1'
    elif x >= tmp[int(3 * len(tmp) / 4)]:
        return 'edu_cat4'
    elif x >= tmp[int(2 * len(tmp) / 4)]:
        return 'edu_cat3'
    else:
        return 'edu_cat2'

df['edu'] = df['edu_orig'].apply(lambda x: cat_edu(x))<jupyter_output><empty_output><jupyter_text>The code above defines the first non-sensitive feature, education and how to convert the feature into categorical.<jupyter_code>temp = []
tmp_list = [(i * 2 - 1) * 0.1 + (j * 2 - 1) * 0.4 + np.log(z + 1) for i, j,
            z in zip(df['sens'].tolist(), df['y'].tolist(), df['edu_orig'].tolist())]
for i in tmp_list:
    temp.append(
        np.random.normal(
            loc=i,
            scale=1.5) +
        np.random.exponential(1) -
        1)
df['occuption_orig'] = temp
tmp = temp


def cat_occ(x):
    if x <= tmp[int(len(tmp) / 4)]:
        return 'occ_cat1'
    elif x >= tmp[int(3 * len(tmp) / 4)]:
        return 'occ_cat4'
    elif x >= tmp[int(2 * len(tmp) / 4)]:
        return 'occ_cat3'
    else:
        return 'occ_cat2'


df['occuption'] = df['occuption_orig'].apply(lambda x: cat_occ(x))


temp = []
tmp_list = [(i * 2 - 1) * 0.05 + (j * 2 -1)*0.05 +np.log(z +1) /2 +
            np.log(np.abs(k)+1)/5 for i,j,z,k in zip(df['sens'].tolist(),
                                                     df['y'].tolist(),
                                                     df['edu_orig'].tolist(),
                                                     df['occuption_orig'].tolist())]
for i in tmp_list:
    temp.append(np.random.normal(loc=i,scale=1.5) + np.random.exponential(1)-1)
df['age_orig'] = temp
tmp = temp
def cat_age(x):
    if x <= tmp[int(len(tmp) / 4)]:
        return 'age_cat1'
    elif x >= tmp[int(3 * len(tmp) / 4)]:
        return 'age_cat4'
    elif x >= tmp[int(2 * len(tmp) / 4)]:
        return 'age_cat3'
    else:
        return 'age_cat2'


df['age'] = df['age_orig'].apply(lambda x: cat_age(x))<jupyter_output><empty_output><jupyter_text>In the part above, we define the other two non-sensitive features. Now, we have finished the synthetic data construction. Next, we will use the synthetic data to do our missing value experiment.<jupyter_code># we first define the distort score metrics to use the categorical fix
def custom_distort(vold, vnew):
    distort = {}
    distort['edu'] = pd.DataFrame(
        {'edu_cat1': [0., 1., 2., 3., 100.],
         'edu_cat2': [1., 0., 1., 2., 100.],
         'edu_cat3': [2., 1., 0., 1., 100.],
         'edu_cat4': [3., 2., 1., 0., 100.],
         'missing': [0., 0., 0., 0., 1.]},
        index=['edu_cat1', 'edu_cat2', 'edu_cat3', 'edu_cat4', 'missing'])
    distort['occuption'] = pd.DataFrame(
        {'occ_cat1': [0., 1., 2., 3.],
         'occ_cat2': [1., 0., 1., 2.],
         'occ_cat3': [2., 1., 0., 1.],
         'occ_cat4': [3., 2., 1., 0.]},
        index=['occ_cat1', 'occ_cat2', 'occ_cat3', 'occ_cat4'])
    distort['age'] = pd.DataFrame(
        {'age_cat1': [0., 1., 2., 3.],
         'age_cat2': [1., 0., 1., 2.],
         'age_cat3': [2., 1., 0., 1.],
         'age_cat4': [3., 2., 1., 0.]},
        index=['age_cat1', 'age_cat2', 'age_cat3', 'age_cat4'])
    distort['sens'] = pd.DataFrame(
        {0.0: [0., 2.],
         1.0: [2., 0.]},
        index=[0.0, 1.0])
    distort['y'] = pd.DataFrame(
        {0.0: [0., 2.],
         1.0: [2., 0.]},
        index=[0.0, 1.0])

    total_cost = 0.0
    for k in vold:
        if k in vnew:
            total_cost += distort[k].loc[vnew[k], vold[k]]

    return total_cost


class CustomDataset(StandardDataset):
    """Adult Census Income Dataset.

    See :file:`aif360/data/raw/adult/README.md`.
    """

    def __init__(self, label_name='y',
                 favorable_classes=['1'],
                 protected_attribute_names=['x_control'],
                 privileged_classes=['0'],
                 instance_weights_name=None,
                 categorical_features=[],
                 features_to_keep=[], features_to_drop=[],
                 na_values=[''], custom_preprocessing=None,
                 df=None,
                 metadata=None):

        super().__init__(
            df=df,
            label_name=label_name,
            favorable_classes=favorable_classes,
            protected_attribute_names=protected_attribute_names,
            privileged_classes=privileged_classes,
            instance_weights_name=instance_weights_name,
            categorical_features=categorical_features,
            features_to_keep=features_to_keep,
            features_to_drop=features_to_drop,
            na_values=na_values,
            custom_preprocessing=custom_preprocessing,
            metadata=metadata)<jupyter_output><empty_output><jupyter_text>The section below is how we create missing values.<jupyter_code>df['y'] = df['y'].astype(int)
df1 = df[['y', 'edu', 'occuption', 'age', 'sens']]
tot = []
for index, row in df1.iterrows():
    result = ''
    for j in df1.columns:
        result = result + str(row[j])
    tot.append(result)
df['tmp_feature'] = tot
df['mis_prob'] = 0
# here, the first element in the column 'tmp_feature' is the label (either 0 or 1 with 0 being negative outcome) 
# and the last element is sensitive value (either 0 or 1, with 0 being unprivileged)
# we define the proportion of missing values in the data. Here the missing values are under MAR
for i in df['tmp_feature'].unique():
    if i[0] == '1' and i[-1] == '0':
        df.loc[df['tmp_feature'] == i, 'mis_prob'] = 0.05
    elif i[-1] == '0':
        df.loc[df['tmp_feature'] == i, 'mis_prob'] = 0.5
    elif i[-1] != '0' and i[0] == '1':
        df.loc[df['tmp_feature'] == i, 'mis_prob'] = 0.05
    else:
        df.loc[df['tmp_feature'] == i, 'mis_prob'] = 0.03
new_label = []
for i, j in zip(df['mis_prob'], df['edu']):
    if np.random.binomial(1, i, 1)[0] == 1:
        new_label.append(-1)
    else:
        new_label.append(j)
df['edu'] = new_label
print('Total number of missing values')
print(len(df.loc[df['edu'] == -1, :].index))
print('Total number of observations')
print(len(df.index))<jupyter_output>Total number of missing values
3970
Total number of observations
20000<jupyter_text>In this section, we create our training and test set for training classifier and validation<jupyter_code>def mod_edu(x):
    if x == -1:
        return 'missing'
    else:
        return x


df['edu'] = df['edu'].apply(lambda x: mod_edu(x))


df_pos = df.loc[df['y'] == 1, :]
df_neg = df.loc[df['y'] == 0, :]

df_train_pos, df_test_pos = train_test_split(
    df_pos, test_size=1000, random_state=10)
df_train_neg, df_test_neg = train_test_split(
    df_neg, test_size=1000, random_state=10)
df_test = df_test_pos.append(df_test_neg)

df_train_tot = df_train_pos.append(df_train_neg)

_, df_train_tot_pos = train_test_split(
    df_train_pos, test_size=4000, random_state=10)
_, df_train_tot_neg = train_test_split(
    df_train_neg, test_size=4000, random_state=10)
df_train = df_train_tot_pos.append(df_train_tot_neg)<jupyter_output><empty_output><jupyter_text>Do the categorical fix using the training set<jupyter_code>orig_cat_train = df_train[['y','age', 'sens', 'occuption', 'edu']]
orig_cat_test = df_test[['y', 'age','sens', 'occuption', 'edu']]

all_protected_attribute_maps = {"sens": {0.0: 0, 1.0: 1}}
D_features = ['sens']
dataset_orig_cat_train = CustomDataset(
    favorable_classes=[1], protected_attribute_names=['sens'], privileged_classes=[
        [1]], categorical_features=[
            'occuption', 'edu','age'], features_to_keep=[
                'occuption', 'edu', 'y', 'sens','age'], df=orig_cat_train, metadata={
                    'label_maps': [{1.0: 1, 0.0: 0}], 'protected_attribute_maps': [
                                all_protected_attribute_maps[x] for x in D_features]})

dataset_orig_cat_test = CustomDataset(
    favorable_classes=[1], protected_attribute_names=['sens'], privileged_classes=[
        [1]], categorical_features=[
            'occuption', 'edu','age'], features_to_keep=[
                'occuption', 'edu', 'y', 'sens','age'], df=orig_cat_test, metadata={
                    'label_maps': [{1.0: 1, 0.0: 0}], 'protected_attribute_maps': [
                                all_protected_attribute_maps[x] for x in D_features]})


privileged_groups = [{'sens': 1}]
unprivileged_groups = [{'sens': 0}]
optim_options = {
    "distortion_fun": custom_distort,
    "epsilon": 0.08,
    "clist": [0.99, 1.99, 2.99],
    "dlist": [.2, 0.1, 0]
}


OP = OptimPreproc(OptTools, optim_options,
                  unprivileged_groups=unprivileged_groups,
                  privileged_groups=privileged_groups)

OP = OP.fit(dataset_orig_cat_train)


dataset_transf_cat_test = OP.transform(dataset_orig_cat_test, transform_Y=True)
dataset_transf_cat_test = dataset_orig_cat_test.align_datasets(
    dataset_transf_cat_test)


dataset_transf_cat_train = OP.transform(
    dataset_orig_cat_train, transform_Y=True)
dataset_transf_cat_train = dataset_orig_cat_train.align_datasets(
    dataset_transf_cat_train)<jupyter_output>Optimized Preprocessing: Objective converged to 0.161315<jupyter_text>We then use the fixed training set to train our logistic regression classifier and get fairness scores<jupyter_code>scale_transf = StandardScaler()
X_train = dataset_orig_cat_train.features
y_train = dataset_orig_cat_train.labels.ravel()

X_test = scale_transf.fit_transform(dataset_orig_cat_test.features)

scale_transf = StandardScaler()
X_train = scale_transf.fit_transform(dataset_transf_cat_train.features)
y_train = dataset_transf_cat_train.labels.ravel()

X_test = scale_transf.fit_transform(dataset_transf_cat_test.features)

lmod = LogisticRegression()
lmod.fit(X_train, y_train)
y_pred = lmod.predict(X_test)
print('After categorical fix without reweight')
get_evaluation(dataset_orig_vt,y_pred,privileged_groups,unprivileged_groups,0,1,1)<jupyter_output>After categorical fix without reweight<jupyter_text>We now apply our fixing algorithm presented in the paper to re-train the logistic regression classifier<jupyter_code>weight_df = dataset_orig_cat_train.convert_to_dataframe()[0]
weight_df['weight'] = 1
weight_df['is_missing'] = 0
weight_df['tmp'] = ''
tmp_result = []
for i, j in zip(weight_df['sens'], weight_df['y']):
    tmp_result.append(str(i) + str(j))
weight_df['tmp'] = tmp_result

weight_df.loc[weight_df['edu=missing'] == 1, 'is_missing'] = 1

for i in weight_df['tmp'].unique():
    weight_df.loc[(weight_df['tmp'] == i) & (weight_df['is_missing'] == 0),
           'weight'] = len(weight_df.loc[(weight_df['tmp'] == i),:].index) / len(weight_df.loc[(weight_df['tmp'] == i) & (weight_df['is_missing'] == 0),:].index)
    weight_df.loc[(weight_df['tmp'] == i) & (weight_df['is_missing'] == 1),
           'weight'] = len(weight_df.loc[(weight_df['tmp'] == i) & (weight_df['is_missing'] == 0),:].index) / len(weight_df.loc[(weight_df['tmp'] == i),:].index)
dataset_orig_cat_train.instance_weights = np.array(weight_df['weight'])


lmod = LogisticRegression()
lmod.fit(X_train, y_train, sample_weight=dataset_orig_cat_train.instance_weights)
y_pred = lmod.predict(X_test)
print('After categorical fix with reweight')
get_evaluation(dataset_orig_vt,y_pred,privileged_groups,unprivileged_groups,0,1,1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1465391355140187, 'fraction_numerical': 0.027599299065420562, 'mean_word_length': 3.4820026178010473, 'pattern_counts': {'":': 5, '<': 41, '<?xml version=': 0, '>': 44, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '270750', 'license': 'MIT', 'n_tokens_mistral': 5026, 'n_tokens_neox': 4734, 'n_words': 1094, 'path': 'Synthetic_data.ipynb', 'repo_name': 'GU-DataLab/fairness-and-missing-values', 'size': '13696'}