@inproceedings{
beckmann2023sam,
title={{SAM} meets Gaze: Passive Eye Tracking for Prompt-based Instance Segmentation},
author={Daniel Beckmann and Jacqueline Kockwelp and Joerg Gromoll and Friedemann Kiefer and Benjamin Risse},
booktitle={NeuRIPS 2023 Workshop on Gaze Meets ML},
year={2023},
url={https://openreview.net/forum?id=o9YkHqBE5I}
}
The annotation of large new datasets for machine learning is a very time-consuming and expensive process. This is particularly true for pixel-accurate labelling of e.g. segmentation masks. Prompt-based methods have been developed to accelerate this label generation process by allowing the model to incorporate additional clues from other sources such as humans. The recently published Segment Anything foundation model (SAM) extends this approach by providing a flexible framework with a model that was trained on more than 1 billion segmentation masks, while also being able to exploit explicit user input. In this paper, we explore the usage of a passive eye tracking system to collect gaze data during unconstrained image inspections which we integrate as a novel prompt input for SAM. We evaluated our method on the original SAM model and finetuned the prompt encoder and mask decoder for different gaze-based inputs, namely fixation points, blurred gaze
maps and multiple heatmap variants. Our results indicate that the acquisition of gaze data is faster than other prompt-based approaches while the segmentation performance stays comparable to the state-of-the-art performance of SAM. Code is available at https://zivgitlab.uni-muenster.de/cvmls/sam_meets_gaze.
SAM meets Gaze: Passive Eye Tracking for Prompt-based Instance Segmentation
Daniel Beckmann Jacqueline Kockwelp Joerg Gromoll Friedemann Kiefer Benjamin Risse