In  [1]}, Serna et al. proposed to study the values of the activations in CNNs for a facial gender recognition task, and discover that when the models have learned a biased representation, the activations in its filters are not as high when dealing with samples from the discriminated groups. In  [2]}, the same group of researchers extended this work by training different groups of NNs, and then used other models to predict the presence of bias from their weights without looking at their inferences, proving that bias is encoded in the model's weights. Other works also interestingly investigate the predictor's hidden activations to detect sub-groups [3]}, [4]}, [5]}, [6]}.
