We measured inter-annotator agreement to measure the clarity of our ACQI taxonomy and IQ rules when applied to real dialog systems.
The results have a Cohen's Kappa (CK) of .68, which is substantial agreement according to [1]}.
See Table REF  for more details.
<TABLE>