However, we want to emphasize that studies focusing on the presence of bias in more general industrial applications based on complex data like images have mostly been left out of the spotlight. We intend to raise awareness about this kind of problem for safety-critical and/or industrial applications, where trained models may be discriminating against a certain group (or situation) in the form of a biased decision or diminished performance. We point out that a team developing a NN-based application might simply be unaware of this behavior until the application is deployed. In this case, specific groups of end-users may observe that it does not work as intended. A typical example of undesired algorithmic bias in image analysis applications is the one that was made popular by the paper presenting the LIME explainability technique [1]}. Indeed, the authors trained a neural network to discriminate images representing wolves and huskies.
Despite the NN's reasonable accuracy, it was still basing itself off spurious correlations – i.e. the presence or not of snow in the background – to decide whether the image contained a wolf. Another example that will be at the heart of this paper is a blue veil effect in satellite images, which will be discussed in Section . When present, these biases provide a shortcut for the models to achieve a higher accuracy score both in the training and test datasets, although the logic behind the decision rules is generally false.
This phenomenon is often modeled by the use of confounding variables in statistics. Hence, they hinder these models' performance when predicting a sample from the discriminated group.
This makes it completely clear that all harmful biases must be addressed in industrial and safety-critical applications, as algorithmic biases might render the general performance guarantees useless in specific or uncommon situations.
