Common metrics for determining inter-annotator agreement, e.g. Cohen's Kappa [1]} or Fleiss' Kappa [2]}, can not be applied when annotating sequences, as the annotators are free to choose which parts of a sequence to include. Therefore, we use the agr metric [3]}, which is defined as:
\( \textrm {agr}(a||b) = \frac{|A \textrm { matching }B|}{|A|}\) 
