When dealing with multi-rater data, most existing methodologies apply simple strategies like majority voting to obtain a unique set of ground-truth labels. However, approaches exploiting multi-rater data more effectively exist and are not new; in their seminal work, [1]} proposed an Expectation-Maximization algorithm to estimate raters' error-rates in multinomial multi-rater data.
More recent works aim at modeling raters' reliability for aggregating or filtering labels, such as [2]} and [3]}. We refer the reader to [4]} for a review of approaches and challenges in inference with multi-rater data.
The recent trend is instead increasingly exploiting multi-rater data, when possible, to increase data efficiency;
in the biomedical context, Wei et al. [5]} proposed a curriculum learning approach on samples with increasing raters' agreement for histopathology image classification. In contrast, Mirikharaji et al. [6]} tackles skin-lesion segmentation by building multiple models (one for each set of raters' labels) and then aggregating models predictions.
To the best of our knowledge, the only proposed counting approach dealing with multi-rater data is [7]}, where authors train a supervised algorithm to count antarctic penguins in images dot-annotated by non-professional volunteers; multi-rater labels are mainly exploited to estimate the object scale, which varies wildly in their dataset (the diameter of a penguin varies between 15 and 700 pixels) and is instead fixed in our scenario.
When dealing with constant scale objects, as in our microscopy images scenario, their solution resembles [8]}, a segmentation-based approach adopted and compared in this work.
Moreover, instead of requiring large multi-rater training sets, our approach is designed to train on a large single-rater set plus a small multi-rater set, lowering the total labeling cost.
