Ranking consistency assesses how consistent a metric is in ranking the methods across the different images. This is measured using Krippendorff's \(\alpha \)  [1]}. Krippendorff's \(\alpha \)  is a statistic usually used to measure inter-rater reliability: the degree to which different raters (for example, for a psychological test) agree in their assessments. Krippendorff's \(\alpha \)  is defined as follows:
\(\alpha = 1 - \frac{D_o}{D_e}\) 
