This leads us to the natural question: do humans and machines process facial expression images differently? And if they do, can we teach machines to act more like humans? In this paper, we adopted a human-in-the-loop (HIL) paradigm to address this question: we first collected click data from human participants to gather those spatial locations that may be important for disambiguating an expression in a two-alternative forced-choice task. Click data are reported to be a cost-efficient substitute for eye-tracking, and reflect the regional attention well [1]}, [2]}. For automatic FER, we trained an ensemble of deep neural networks on AffectNet [3]}, a large, in-the-wild facial expression dataset, and compared its activation map with human click data using three different explainability or visualization methods. We also tried to further fine-tune the models with the human attention map to see whether this would improve the FER performance.
