<jupyter_start><jupyter_text>Fairness in IBM FL: Scikitlearn Logistic Classification  Outline:- [Add conda environment to Jupyter Notebook](setup)- [FL and Fairness](intro)- [Parties](Parties)    - [Party Configuration](Party-Configuration)    - [Party Setup](Party-Setup)- [Register All Parties Before Starting Training](Register-All-Parties-Before-Starting-Training)- [Visualize Results](Visualize-Results)- [Shut Down](Shut-Down)  Add conda environment to Jupyter Notebook Please ensure that you have activated the `conda` environment following the instructions in the project README.Once done, run the following commands in your terminal to install your conda environment into the Jupyter Notebook:1. Once you have activated the conda environment, install the `ipykernel` package: `conda install -c anaconda ipykernel`2. Next, install the `ipykernel` module within Jupyter Notebook: `python -m ipykernel install --user --name=`3. Finally, restart the jupyter notebook once done. Ensure that you are running this Notebook from `/examples/sklearn_logclassification`, where project_path is the directory where the IBMFL repository was cloned.When the Notebook is up and running it may prompt you to choose the kernel. Use the drop down to choose the kernel name same as that chosen when running `conda activate `. If no prompt shows up, you can change the kernel by clicking _Kernel_ > _Change kernel_ > _``_.  Federated Learning (FL) **Federated Learning (FL)** is a distributed machine learning process in which each participant node (or party) retains their data locally and interacts with  other participants via a learning protocol. In this notebook, we demonstrate the adaption and usage of popular bias mitigation techniques for FL. We examine bias from the perspective of social fairness, as opposed to contribution fairness.Bias mitigation approaches in machine learning mainly measure and reduce undesired bias with respect to a *sensitive attribute*, such as *age* or *race*, in the training dataset.We utilize [IBM FL](https://github.com/IBM/federated-learning-lib) to have multiple parties train a classifier to predict whether a person in the [Adult dataset](http://archive.ics.uci.edu/ml/datasets/Adult) makes over $50,000 a year. We have adapted 2 centralized fairness methods, Reweighing and Prejudice Remover, into 3 federated learning bias mitigation methods: Local Reweighing, Global Reweighing with Differential Privacy, and Federated Prejudice Removal.For a more technical dive into IBM FL, refer the whitepaper [here](https://arxiv.org/pdf/2007.10987.pdf).  Fairness Techniques We adapt a centralized pre-processing bias mitigation method [Reweighing](https://link.springer.com/article/10.1007/s10115-011-0463-8) into two federated learning techniques, Local Reweighing and Global Reweighing with Differential Privacy.**Local reweighing**: To fully protect parties' data privacy, each party computes reweighing weights locally based on its own training dataset during pre-processing and then uses the reweighing dataset for its local training. Therefore, parties do not need to communicate with the aggregator or reveal their sensitive attributes and data sample information.**Global Reweighing with Differential Privacy**: If parties agree to share sensitive attributes and noisy data statistics, parties can employ this fairness method. During the pre-processing phase, the aggregator will collect statistics such as the noisy number of samples with privileged attribute values, compute global reweighing weights  based on the collected statistics, and share them with parties. By adjusting the amount of noise injected via epsilon, parties can control their data leakage while still mitigating bias. We also adapt an in-processing bias mitigation method into Federated Prejudice Remover.**Federated Prejudice Removal**: Each party applies the [Prejudice Remover algorithm](https://github.com/algofairness/fairness-comparison/tree/master/fairness/algorithms/kamishima) to train a less biased local model, and shares only the model parameters with the aggregator. The aggregator can then employ existing FL algorithms, like simple average and FedAvg, etc., to update the global model.Further details about the algorithms and datasets utilized, as well as experimental setup, are included in the paper titled [Mitigating Bias in Federated Learning](https://arxiv.org/abs/2012.02447).  Fairness Metrics In fairness evaluation, there is no single, all-inclusive metric. Literature uses multiple metrics to measure several aspects, painting a composition of fairness. We use four highly-utilized fairness metrics: Statistical Parity Difference, Equal Odds Difference, Average Odds Difference, and Disparate Impact.**Statistical Parity Difference**: Calculated as the ratio of the success rate between the unprivileged and privileged groups. The ideal value for this metric is 0, and the fairness range is between -0.1 and 0.1, as defined by [AI Fairness 360](https://aif360.mybluemix.net/).**Equal Odds Difference**: Calculated as the true positive rate difference between the unprivileged and privileged groups. The ideal value for this metric is 0, and the fairness range is between -0.1 and 0.1, similarly defined by AI Fairness 360.**Average Odds Difference**: Calculated as the mean of the false positive rate difference and the true positive rate difference, both between the unprivileged and privileged groups. The ideal value for this metric is 0, and the fairness range is between -0.1 and 0.1, similarly defined by AI Fairness 360.**Disparate Impact**: Calculated as the difference of the success rate between the unprivileged and privileged groups. The ideal value for this metric is 1, and the fairness range is between 0.8 and 1.2, similarly defined by AI Fairness 360.  Getting things readyWe begin by setting the number of parties that will participate in the federated learning run and splitting up the data among them.<jupyter_code>import sys
party_id = 0

sys.path.append('../..')
import os
os.chdir("../..")

num_parties = 2  ## number of participating parties
dataset = 'adult'<jupyter_output><empty_output><jupyter_text>PartiesEach party holds its own dataset that is kept to itself and used to answer queries received from the aggregator. Because each party may have stored data in different formats, FL offers an abstraction called Data Handler. This module allows for custom implementations to retrieve the data from each of the participating parties. A local training handler sits at each party to control the local training happening at the party side.   Party Configuration**Note**: in a typical FL setting, the parties may have very different configurations from each other. However, in this simplified example, the config does not differ much across parties. So, we first setup the configuration common to both parties, in the next cell. We discuss the parameters that are specific to each, in subsequent cells. Image Source: IBM Federated Learning: An Enterprise FrameworkWhite Paper V0.1  Party SetupIn the following cell, we setup configurations for parties, including network-level details, hyperparameters as well as the model specifications. Please note that if you are running this notebook in distributed environment on separate nodes then you need to split the data locally and obtain the model h5 generated by the Aggregator.<jupyter_code>def get_party_config(party_id):
    party_config = {
        'aggregator':
            {
                'ip': '127.0.0.1',
                'port': 5000
            },
        'connection': {
            'info': {
                'ip': '127.0.0.1',
                'port': 8085 + party_id,
                'id': 'party' + str(party_id),
                'tls_config': {
                    'enable': False
                }
            },
            'name': 'FlaskConnection',
            'path': 'ibmfl.connection.flask_connection',
            'sync': False
        },
        'data': {
            'info': {
                'txt_file': 'examples/data/adult/random/data_party'+ str(party_id) +'.csv'
            },
            'name': 'AdultSklearnDataHandler',
            'path': 'ibmfl.util.data_handlers.adult_sklearn_data_handler'
        },
        'local_training': {
            'name': 'ReweighLocalTrainingHandler',
            'path': 'ibmfl.party.training.reweigh_local_training_handler'
        },
        'model': {
            'name': 'SklearnSGDFLModel',
            'path': 'ibmfl.model.sklearn_SGD_linear_fl_model',
            'spec': {
                'model_definition': 'examples/configs/sklearn_logclassification_rw/model_architecture.pickle',
            }
        },
        'protocol_handler': {
            'name': 'PartyProtocolHandler',
            'path': 'ibmfl.party.party_protocol_handler'
        }
    }
    return party_config<jupyter_output><empty_output><jupyter_text>Running the PartyNow, we invoke the `get_party_config` function to setup party and `start()` it.Finally, we register the party with the Aggregator.<jupyter_code>from ibmfl.party.party import Party
import tensorflow as tf

party_config = get_party_config(party_id)
party = Party(config_dict=party_config)
party.start()
party.register_party()
party.proto_handler.is_private = False  ## allows sharing of metrics with aggregator<jupyter_output>2021-02-04 13:31:04,612 | 1.0.3 | INFO | ibmfl.util.config                             | Getting Aggregator details from arguments.<jupyter_text>Register All Parties Before Starting TrainingNow we have started and registered this Party. Next, we will start and register rest of the parties. Once all the Parties have registered we will go back to the Aggregator's notebook to start training.  ResultsWe utilize these methods in our paper, and share below our experimental results for Local Reweighing and Federated Prejudice Remover on the Adult Dataset. Using 8 parties and a global testing set, we find both methods to be effective in reducing bias as measured by these four fairness metrics. Local reweighing is particularly effective.Fairness Metrics for Local Reweighing and Federated Prejudice Remover on 8 Party Adult Dataset Experiment  Shut DownInvoke the `stop()` method on each of the network participants to terminate the service.<jupyter_code>party.stop()<jupyter_output>2020-10-03 23:07:38,861 -STD ibmfl.connection.flask_connection - INFO - Stopping Receiver and Sender
2020-10-03 23:07:38,867 -STD werkzeug - INFO - 127.0.0.1 - - [03/Oct/2020 23:07:38] "[37mPOST /shutdown HTTP/1.1[0m" 200 -
2020-10-03 23:07:38,869 -STD ibmfl.party.party - INFO - Party stop successful
{'fraction_non_alphanumeric': 0.06807710390220968, 'fraction_numerical': 0.01937000470145745, 'mean_word_length': 4.1159211159211155, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 17, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '416793', 'license': 'IBM-pibs', 'n_tokens_mistral': 2868, 'n_tokens_neox': 2613, 'n_words': 1242, 'path': 'Notebooks/sklearn_logclassification_rw/sklearn_logclassification_rw_p0.ipynb', 'repo_name': 'ch4174nya/federated-learning-lib', 'size': '10635'}