Neural Networks (NNs) have become an increasingly central component of modern decision-making systems, including those that are used in sensitive/legal domains such as crime prediction [1]}, credit assessment [2]}, income prediction [2]}, and hiring decisions. However, studies have shown that these systems are prone to biases [4]} that deem their usage unfair to unprivileged users based on their age, race, or gender. The bias is usually either inherent in the training data or introduced during the training process. Mitigating algorithmic bias has been studied in the literature  [5]}, [6]}, [7]} in the context of group and individual fairness. However, the fairness of the NN is considered only empirically on the test data with the hope that it represents the underlying data distribution.
