license: mit
MSI-Net
Contextual encoder-decoder network for visual saliency prediction
Summary
MSI-Net is a visual saliency model that predicts where humans fixate on natural images using a contextual encoder-decoder network trained on eye movement data. The model is based on a convolutional neural network architecture and includes an ASPP module with multiple convolutional layers at different dilation rates to capture multi-scale features in parallel. Moreover, it combines the resulting representations with global scene information towards accurate predictions of visual saliency. MSI-Net consists of roughly 25M parameters and thus presents a suitable choice for applications with limited computational resources. For more information on the model, check out GitHub and the corresponding paper or preprint.
Datasets
Before training the model on fixation data, the encoder weights were initialized from a VGG16 backbone pre-trained on the ImageNet classification task. The model was then trained on the SALICON dataset, which consists of mouse movement recordings as a proxy for gaze measurements. Finally, the weights can be fine-tuned on human eye tracking data. MSI-Net was therefore also trained on one of the following datasets, although here we only provide the SALICON base model:
| Number of Images | Viewers per Image | Viewing Duration | Recording Type | |
|---|---|---|---|---|
| SALICON | 10,000 | 16 | 5s | Mouse tracking |
| MIT1003 | 1,003 | 15 | 3s | Eye tracking |
| CAT2000 | 4,000 | 24 | 5s | Eye tracking |
| DUT-OMRON | 5,168 | 5 | 2s | Eye tracking |
| PASCAL-S | 850 | 8 | 2s | Eye tracking |
| OSIE | 700 | 15 | 3s | Eye tracking |
| FIWI | 149 | 11 | 5s | Eye tracking |
Evaluations of our model are available on the original MIT saliency benchmark and the updated MIT/Tübingen saliency benchmark. Results for the latter are derived from a probabilistic representation of predicted saliency maps with metric-specific postprocessing for a fair model comparison.
Limitations
MSI-Net was trained on human fixation data collected under a free-viewing paradigm. Therefore, the predicted saliency maps may not generalize to viewers that received task instructions during the experiment. It must also be noted that the training data consisted primarily of natural images. As a result, gaze predictions for specific image types (e.g., fractals, patterns) or adversarial examples may not be very accurate.
Another limitation is that saliency-based cropping algorithms, formerly applied to images uploaded to the social media platform Twitter between 2018 and 2021, have shown biases in terms of race and gender. It is thus important to use saliency models with caution and acknowledge the potential risks that are involved in their application.
Reference
If you find this code or model useful, please cite the following paper:
@article{kroner2020contextual,
title={Contextual encoder-decoder network for visual saliency prediction},
author={Kroner, Alexander and Senden, Mario and Driessens, Kurt and Goebel, Rainer},
url={http://www.sciencedirect.com/science/article/pii/S0893608020301660},
doi={https://doi.org/10.1016/j.neunet.2020.05.004},
journal={Neural Networks},
publisher={Elsevier},
year={2020},
volume={129},
pages={261--270},
issn={0893-6080}
}