EC1 Reliability.
One aspect when working with correlation values is the trustworthiness of a bias calculation.
Correlation values are always dependant on the sample size they were calculated on [1]}.
If there are too few samples to base the metric calculation on, resulting metric values might not represent real data distributions well, and, thus, need to be double-checked.
A small number of samples for a feature or sensitive attribute can in fact be a warning sign in itself, indicating that either more data is needed, or the model is particularly bad at recognizing said feature.
Domain experts should, therefore, be able to access additional information about the correlation, to help them assess whether a correlation value is supported by representative data.
While visualizations based on classical fairness metrics do not face this challenge, it needs to be addressed by our approach in that such potential problems need to be visually communicated.
