When there are multiple reference segmentations available, a similarity measure index can be obtained by comparing an automatic segmentation with the set of references [1]}. In medical imaging, the problem of performance analysis with multiple references which may suffer from intra-rater and inter-rater variability, has been addressed [2]}, [3]}. The STAPLE approach [2]} has lead to the work of Bouix et al. [5]} that proposed techniques for comparing the relative performance of different methods without the need of GT. Here, the different segmentation results are treated as plausible references, thus can be evaluated through STAPLE and the concept of common agreement. Another work by [6]} has quantitatively evaluated the performance of several segmentation algorithms by region-correlation matrix. The limitation of this work is that it cannot evaluate the segmentation performance of a particular method on a particular image independently.
