For the annotation of action mention extents, which is the most important step in our re-annotation effort, we reach 0.80 \(\alpha _U\) , indicating good reliability between annotators [1]}, [2]}.
The agreement for the annotation of participants, time, and location is lower at 0.57 \(\alpha _U\) . We found that this mostly stems from the annotation of participants. In our guidelines, we specify that annotators should only mark an entity as a participant of an event if it plays a significant role in the event action. The larger and more coarse an event is, the more difficult this decision becomes for annotators. One such case is shown in example:difficultannotation where it is debatable if "Christian Teinturier" is or is not significantly involved in the tournament event.

"Earlier today, French Football Federation vice-president Christian Teinturier said if there was any basis to the reports about Anelka then he should be sent home from the tournament immediately."


A second reason is that we do not annotate entity coreference, so only a single entity mention is meant to be annotated for each entity participating in an event. In case the same entity appears twice in a sentence, we instruct annotators to choose the more specific description. If the candidates are identical in surface form, annotators are meant to choose the one closer (in word distance) to the event action. There remains a level of subjectivity in these decisions, leading to disagreement.
