During the user study, we focused on nPMI as a fairness metric and used open-sourced predictions associated with OID [1]}, which was randomly split into two subsets (60%/40%).
All predictions and visualizations were based on the whole OID label set (about 20,000 labels).
We calculated nPMI scores for all annotations with respect to the sensitive attribute directions male and female.
Then, we used our TensorBoard plugin to present the proposed visualization approach to our participants.
