By exploiting the widely known GradCAM method [1]}, Tong and Kagal [2]} recover the image classification model properties when making a decision. Their intuition is that the results could expose biases learned by the model.
For example, in a dataset where most of doctors are males, GradCAM exposed the fact that the model’s predictions focused mainly on the facial features of the person, while clothes and accessories are highlighted for female doctors. The same conclusions were drawn for basketball players, where GradCAM explained that the predictions were mainly based on the players' faces and not on basketball‐related features. Although the predictions were often accurate, explanations made indeed clear that they were based on players’ faces because the training dataset contained a lot of female volleyball players, and facial features help a lot to predict a person gender. Finally, Schaaf et al. [3]} combined attribution methods with ground truth masks to help detecting biases.
