Finally, we can discuss the results. We can first notice that the re-weighting technique of [1]} had little effect on the results. It was indeed designed to correct bias that manifests in the form of disparate impact, so it did not reduce the error rate gap between groups. The debiasing method must then be specifically chosen to target the bias through the metric with which it was measured.
Using the pre-trained network of Torchvision had a disastrous effect when only optimizing the last neural-network layer, but was particularly efficient when using fine-tuning, i.e. when simultaneously optimizing the transformation of the data representation and the final decision rules. Using a relevant initial state, when available, and using fine tuning then appears as a very good strategy here. It is often denoted by transfer learning in the machine learning literature.
Interestingly, similar results were obtained with a random initial state, when stopping the training procedure at an iteration where the trained neural-network had good generalization properties on the blue-veiled images. Understanding this result requires to look closely at the convergence curves, as illustrated Fig. REF  on a typical run:
