Here, we model agreement by learning to rank a tuple of samples with increasing agreement values.
Our formulation instantiates a standard pairwise learning to rank approach [1]} with a custom sample loss definition.
Specifically, we still define a model with a scalar output \(s = g_\phi (o)\) , but we employ a different training scheme;
given a \((K+1)\) -tuple of ordered samples \(O = (o_0, \dots , o_K)\)  containing one sample per agreement class (i.e., \(o_i\)  has an agreement value \(a_i = i\) ), we ask our model to produce scores \(s_i = g_\phi (o_i)\)  that are sorted \(s_0 < s_1 < \dots < s_K\) .
Translating this constraint in a loss function for the single tuple, we obtain a class-balanced pairwise margin loss
\(\mathcal {L}(o_0, \dots , o_K; \phi ) = \frac{1}{K} \sum _{i=1}^{K} \max ( m - g_\phi (o_i) + g_\phi (o_{i-1}), 0 )\,,\) 
