natural language toolkit agreement metrics c 20012023 nltk project tom lippincott tomcs columbia edu url https www nltk org for license information see license txt implementations of interannotator agreement coefficients surveyed by artstein and poesio 2007 intercoder agreement for computational linguistics an agreement coefficient calculates the amount that annotators agreed on label assignments beyond what is expected by chance in defining the annotationtask class we use naming conventions similar to the paper s terminology there are three types of objects in an annotation task the coders variables c and c the items to be annotated variables i and i the potential categories to be assigned variables k and k additionally it is often the case that we don t want to treat two different labels as complete disagreement and so the annotationtask constructor can also take a distance metric as a final argument distance metrics are simply functions that take two arguments and return a value between 0 0 and 1 0 indicating the distance between them if not supplied the default is binary comparison between the arguments the simplest way to initialize an annotationtask is with a list of triples each containing a coder s assignment for one object in the task task annotationtaskdata c1 1 v1 c2 1 v1 note that the data list needs to contain the same number of triples for each individual coder containing category values for the same set of items alpha krippendorff 1980 kappa cohen 1960 s bennet albert and goldstein 1954 pi scott 1955 todo describe handling of multiple coders and missing data expected results from the artstein and poesio survey paper from nltk metrics agreement import annotationtask import os path t annotationtaskdatax split for x in openos path joinos path dirnamefile artsteinpoesioexample txt t avgao 0 88 roundt pi 5 0 79953 roundt s 2 0 82 this would have returned a wrong value 0 0 in 785fb79 as coders are in the wrong order subsequently all values for pi s and kappa would have been wrong as they are computed with avgao t2 annotationtaskdata b 1 stat a 1 stat t2 avgao 1 0 the following of course also works t3 annotationtaskdata a 1 othr b 1 othr t3 avgao 1 0 represents an annotation task i e people assign labels to items notation tries to match notation in artstein and poesio 2007 in general coders and items can be represented as any hashable object integers for example are fine though strings are more readable labels must support the distance functions applied to them so e g a stringeditdistance makes no sense if your labels are integers whereas interval distance needs numeric values a notable case of this is the masi metric which requires python sets initialize an annotation task the data argument can be none to create an empty annotation task or a sequence of 3tuples each representing a coder s labeling of an item coder item label the distance argument is a function taking two arguments labels and producing a numerical distance the distance from a label to itself should be zero distancel l 0 load an sequence of annotation results appending to any data already loaded the argument is a sequence of 3tuples each representing a coder s labeling of an item coder item label agreement between two coders on a given item data data or self data cfedermann we don t know what combination of coderitem will come first in x to avoid stopiteration problems due to assuming an order ca cb we allow either for k1 and then look up the missing as k2 k1 nextx for x in data if xcoder in ca cb and xitem i if k1coder ca k2 nextx for x in data if xcoder cb and xitem i else k2 nextx for x in data if xcoder ca and xitem i ret 1 0 floatself distancek1labels k2labels log debugobserved agreement between s and s on s f ca cb i ret log debug distance between r and r f k1labels k2labels 1 0 ret return ret def nkself k return floatsum1 for x in self data if xlabels k def nikself i k return floatsum1 for x in self data if xitem i and xlabels k def nckself c k return floatsum1 for x in self data if xcoder c and xlabels k deprecateduse nk nik or nck instead def nself knone inone cnone observed agreement between two coders on all items data self groupeddata item x for x in self data if xcoder in ca cb ret sumself agrca cb item itemdata for item itemdata in data len self i log debugobserved agreement between s and s f ca cb ret return ret def pairwiseaverageself function total 0 n 0 s self c copy for ca in self c s removeca for cb in s total functionca cb n 1 ret total n return ret def avgaoself the observed disagreement for the weighted kappa coefficient total 0 0 data x for x in self data if xcoder in ca cb for i itemdata in self groupeddataitem data we should have two items distance doesn t care which comes first total self distancenextitemdatalabels nextitemdatalabels ret total lenself i maxdistance log debugobserved disagreement between s and s f ca cb ret return ret def dokwself maxdistance1 0 agreement coefficients bennett albert and goldstein 1954 ae 1 0 lenself k ret self avgao ae 1 0 ae return ret def piself total 0 0 labelfreqs freqdistxlabels for x in self data for k f in labelfreqs items total f2 ae total lenself i lenself c 2 return self avgao ae 1 ae def aekappaself ca cb ae 0 0 nitems floatlenself i labelfreqs conditionalfreqdistxlabels xcoder for x in self data for k in labelfreqs conditions ae labelfreqskca nitems labelfreqskcb nitems return ae def kappapairwiseself ca cb cohen 1960 averages naively over kappas for each coder pair davies and fleiss 1982 averages over observed and expected agreements for each coder pair krippendorff 1980 check for degenerate cases if lenself k 0 raise valueerrorcannot calculate alpha no data present if lenself k 1 log debugonly one annotation value alpha returning 1 return 1 if lenself c 1 and lenself i 1 raise valueerrorcannot calculate alpha only one coder and item present totaldisagreement 0 0 totalratings 0 allvalidlabelsfreq freqdist totaldo 0 0 total observed disagreement for all items for i itemdata in self groupeddataitem labelfreqs freqdistxlabels for x in itemdata labelscount sumlabelfreqs values if labelscount 2 ignore the item continue allvalidlabelsfreq labelfreqs totaldo self disagreementlabelfreqs labelscount do totaldo sumallvalidlabelsfreq values de self disagreementallvalidlabelsfreq expected disagreement kalpha 1 0 do de return kalpha def weightedkappapairwiseself ca cb maxdistance1 0 cohen 1968 return self pairwiseaverage lambda ca cb self weightedkappapairwiseca cb maxdistance if name main import optparse import re from nltk metrics import distance process commandline arguments parser optparse optionparser parser addoption d distance destdistance defaultbinarydistance helpdistance metric to use parser addoption a agreement destagreement defaultkappa helpagreement coefficient to calculate parser addoption e exclude destexclude actionappend default helpcoder names to exclude may be specified multiple times parser addoption i include destinclude actionappend default helpcoder names to include same format as exclude parser addoption f file destfile helpfile to read labelings from each line with three columns labeler item labels parser addoption v verbose destverbose default0 helphow much debugging to print on stderr 04 parser addoption c columnsep destcolumnsep defaultt helpcharstring that separates the three columns in the file defaults to tab parser addoption l labelsep destlabelsep default helpcharstring that separates labels if labelers can assign more than one defaults to comma parser addoption p presence destpresence defaultnone helpconvert each labeling into 1 or 0 based on presence of label parser addoption t thorough destthorough defaultfalse actionstoretrue helpcalculate agreement for every subset of the annotators options remainder parser parseargs if not options file parser printhelp exit logging basicconfiglevel50 10 intoptions verbose read in data from the specified file data with openoptions file as infile for l in infile toks l splitoptions columnsep coder object labels toks0 strtoks1 1 frozensettoks1 strip splitoptions labelsep if options include options exclude or lenoptions include 0 and coder in options include or lenoptions exclude 0 and coder not in options exclude data appendcoder object labels if options presence task annotationtask data getattrdistance options distanceoptions presence else task annotationtaskdata getattrdistance options distance if options thorough pass else printgetattrtask options agreement logging shutdown natural language toolkit agreement metrics c 2001 2023 nltk project tom lippincott tom cs columbia edu url https www nltk org for license information see license txt implementations of inter annotator agreement coefficients surveyed by artstein and poesio 2007 inter coder agreement for computational linguistics an agreement coefficient calculates the amount that annotators agreed on label assignments beyond what is expected by chance in defining the annotationtask class we use naming conventions similar to the paper s terminology there are three types of objects in an annotation task the coders variables c and c the items to be annotated variables i and i the potential categories to be assigned variables k and k additionally it is often the case that we don t want to treat two different labels as complete disagreement and so the annotationtask constructor can also take a distance metric as a final argument distance metrics are simply functions that take two arguments and return a value between 0 0 and 1 0 indicating the distance between them if not supplied the default is binary comparison between the arguments the simplest way to initialize an annotationtask is with a list of triples each containing a coder s assignment for one object in the task task annotationtask data c1 1 v1 c2 1 v1 note that the data list needs to contain the same number of triples for each individual coder containing category values for the same set of items alpha krippendorff 1980 kappa cohen 1960 s bennet albert and goldstein 1954 pi scott 1955 todo describe handling of multiple coders and missing data expected results from the artstein and poesio survey paper from nltk metrics agreement import annotationtask import os path t annotationtask data x split for x in open os path join os path dirname __file__ artstein_poesio_example txt t avg_ao 0 88 round t pi 5 0 79953 round t s 2 0 82 this would have returned a wrong value 0 0 in 785fb79 as coders are in the wrong order subsequently all values for pi s and kappa would have been wrong as they are computed with avg_ao t2 annotationtask data b 1 stat a 1 stat t2 avg_ao 1 0 the following of course also works t3 annotationtask data a 1 othr b 1 othr t3 avg_ao 1 0 represents an annotation task i e people assign labels to items notation tries to match notation in artstein and poesio 2007 in general coders and items can be represented as any hashable object integers for example are fine though strings are more readable labels must support the distance functions applied to them so e g a string edit distance makes no sense if your labels are integers whereas interval distance needs numeric values a notable case of this is the masi metric which requires python sets initialize an annotation task the data argument can be none to create an empty annotation task or a sequence of 3 tuples each representing a coder s labeling of an item coder item label the distance argument is a function taking two arguments labels and producing a numerical distance the distance from a label to itself should be zero distance l l 0 load an sequence of annotation results appending to any data already loaded the argument is a sequence of 3 tuples each representing a coder s labeling of an item coder item label agreement between two coders on a given item cfedermann we don t know what combination of coder item will come first in x to avoid stopiteration problems due to assuming an order ca cb we allow either for k1 and then look up the missing as k2 implements the n notation used in artstein and poesio 2007 observed agreement between two coders on all items calculates the average of function results for each coder pair average observed agreement across all coders and items the observed disagreement for the weighted kappa coefficient we should have two items distance doesn t care which comes first averaged over all labelers agreement coefficients bennett albert and goldstein 1954 scott 1955 here multi pi equivalent to k from siegel and castellan 1988 cohen 1960 averages naively over kappas for each coder pair davies and fleiss 1982 averages over observed and expected agreements for each coder pair krippendorff 1980 check for degenerate cases total observed disagreement for all items ignore the item expected disagreement cohen 1968 cohen 1968 process command line arguments read in data from the specified file
import logging
from itertools import groupby
from operator import itemgetter

from nltk.internals import deprecated
from nltk.metrics.distance import binary_distance
from nltk.probability import ConditionalFreqDist, FreqDist

log = logging.getLogger(__name__)


class AnnotationTask:
    

    def __init__(self, data=None, distance=binary_distance):
        
        self.distance = distance
        self.I = set()
        self.K = set()
        self.C = set()
        self.data = []
        if data is not None:
            self.load_array(data)

    def __str__(self):
        return "\r\n".join(
            map(
                lambda x: "%s\t%s\t%s"
                % (x["coder"], x["item"].replace("_", "\t"), ",".join(x["labels"])),
                self.data,
            )
        )

    def load_array(self, array):
        
        for coder, item, labels in array:
            self.C.add(coder)
            self.K.add(labels)
            self.I.add(item)
            self.data.append({"coder": coder, "labels": labels, "item": item})

    def agr(self, cA, cB, i, data=None):
        
        data = data or self.data
        
        
        
        k1 = next(x for x in data if x["coder"] in (cA, cB) and x["item"] == i)
        if k1["coder"] == cA:
            k2 = next(x for x in data if x["coder"] == cB and x["item"] == i)
        else:
            k2 = next(x for x in data if x["coder"] == cA and x["item"] == i)

        ret = 1.0 - float(self.distance(k1["labels"], k2["labels"]))
        log.debug("Observed agreement between %s and %s on %s: %f", cA, cB, i, ret)
        log.debug(
            'Distance between "%r" and "%r": %f', k1["labels"], k2["labels"], 1.0 - ret
        )
        return ret

    def Nk(self, k):
        return float(sum(1 for x in self.data if x["labels"] == k))

    def Nik(self, i, k):
        return float(sum(1 for x in self.data if x["item"] == i and x["labels"] == k))

    def Nck(self, c, k):
        return float(sum(1 for x in self.data if x["coder"] == c and x["labels"] == k))

    @deprecated("Use Nk, Nik or Nck instead")
    def N(self, k=None, i=None, c=None):
        
        if k is not None and i is None and c is None:
            ret = self.Nk(k)
        elif k is not None and i is not None and c is None:
            ret = self.Nik(i, k)
        elif k is not None and c is not None and i is None:
            ret = self.Nck(c, k)
        else:
            raise ValueError(
                f"You must pass either i or c, not both! (k={k!r},i={i!r},c={c!r})"
            )
        log.debug("Count on N[%s,%s,%s]: %d", k, i, c, ret)
        return ret

    def _grouped_data(self, field, data=None):
        data = data or self.data
        return groupby(sorted(data, key=itemgetter(field)), itemgetter(field))

    def Ao(self, cA, cB):
        
        data = self._grouped_data(
            "item", (x for x in self.data if x["coder"] in (cA, cB))
        )
        ret = sum(self.agr(cA, cB, item, item_data) for item, item_data in data) / len(
            self.I
        )
        log.debug("Observed agreement between %s and %s: %f", cA, cB, ret)
        return ret

    def _pairwise_average(self, function):
        
        total = 0
        n = 0
        s = self.C.copy()
        for cA in self.C:
            s.remove(cA)
            for cB in s:
                total += function(cA, cB)
                n += 1
        ret = total / n
        return ret

    def avg_Ao(self):
        
        ret = self._pairwise_average(self.Ao)
        log.debug("Average observed agreement: %f", ret)
        return ret

    def Do_Kw_pairwise(self, cA, cB, max_distance=1.0):
        
        total = 0.0
        data = (x for x in self.data if x["coder"] in (cA, cB))
        for i, itemdata in self._grouped_data("item", data):
            
            total += self.distance(next(itemdata)["labels"], next(itemdata)["labels"])

        ret = total / (len(self.I) * max_distance)
        log.debug("Observed disagreement between %s and %s: %f", cA, cB, ret)
        return ret

    def Do_Kw(self, max_distance=1.0):
        
        ret = self._pairwise_average(
            lambda cA, cB: self.Do_Kw_pairwise(cA, cB, max_distance)
        )
        log.debug("Observed disagreement: %f", ret)
        return ret

    
    def S(self):
        
        Ae = 1.0 / len(self.K)
        ret = (self.avg_Ao() - Ae) / (1.0 - Ae)
        return ret

    def pi(self):
        
        total = 0.0
        label_freqs = FreqDist(x["labels"] for x in self.data)
        for k, f in label_freqs.items():
            total += f**2
        Ae = total / ((len(self.I) * len(self.C)) ** 2)
        return (self.avg_Ao() - Ae) / (1 - Ae)

    def Ae_kappa(self, cA, cB):
        Ae = 0.0
        nitems = float(len(self.I))
        label_freqs = ConditionalFreqDist((x["labels"], x["coder"]) for x in self.data)
        for k in label_freqs.conditions():
            Ae += (label_freqs[k][cA] / nitems) * (label_freqs[k][cB] / nitems)
        return Ae

    def kappa_pairwise(self, cA, cB):
        
        Ae = self.Ae_kappa(cA, cB)
        ret = (self.Ao(cA, cB) - Ae) / (1.0 - Ae)
        log.debug("Expected agreement between %s and %s: %f", cA, cB, Ae)
        return ret

    def kappa(self):
        
        return self._pairwise_average(self.kappa_pairwise)

    def multi_kappa(self):
        
        Ae = self._pairwise_average(self.Ae_kappa)
        return (self.avg_Ao() - Ae) / (1.0 - Ae)

    def Disagreement(self, label_freqs):
        total_labels = sum(label_freqs.values())
        pairs = 0.0
        for j, nj in label_freqs.items():
            for l, nl in label_freqs.items():
                pairs += float(nj * nl) * self.distance(l, j)
        return 1.0 * pairs / (total_labels * (total_labels - 1))

    def alpha(self):
        
        
        if len(self.K) == 0:
            raise ValueError("Cannot calculate alpha, no data present!")
        if len(self.K) == 1:
            log.debug("Only one annotation value, alpha returning 1.")
            return 1
        if len(self.C) == 1 and len(self.I) == 1:
            raise ValueError("Cannot calculate alpha, only one coder and item present!")

        total_disagreement = 0.0
        total_ratings = 0
        all_valid_labels_freq = FreqDist([])

        total_do = 0.0  
        for i, itemdata in self._grouped_data("item"):
            label_freqs = FreqDist(x["labels"] for x in itemdata)
            labels_count = sum(label_freqs.values())
            if labels_count < 2:
                
                continue
            all_valid_labels_freq += label_freqs
            total_do += self.Disagreement(label_freqs) * labels_count

        do = total_do / sum(all_valid_labels_freq.values())

        de = self.Disagreement(all_valid_labels_freq)  
        k_alpha = 1.0 - do / de

        return k_alpha

    def weighted_kappa_pairwise(self, cA, cB, max_distance=1.0):
        
        total = 0.0
        label_freqs = ConditionalFreqDist(
            (x["coder"], x["labels"]) for x in self.data if x["coder"] in (cA, cB)
        )
        for j in self.K:
            for l in self.K:
                total += label_freqs[cA][j] * label_freqs[cB][l] * self.distance(j, l)
        De = total / (max_distance * pow(len(self.I), 2))
        log.debug("Expected disagreement between %s and %s: %f", cA, cB, De)
        Do = self.Do_Kw_pairwise(cA, cB)
        ret = 1.0 - (Do / De)
        return ret

    def weighted_kappa(self, max_distance=1.0):
        
        return self._pairwise_average(
            lambda cA, cB: self.weighted_kappa_pairwise(cA, cB, max_distance)
        )


if __name__ == "__main__":

    import optparse
    import re

    from nltk.metrics import distance

    
    parser = optparse.OptionParser()
    parser.add_option(
        "-d",
        "--distance",
        dest="distance",
        default="binary_distance",
        help="distance metric to use",
    )
    parser.add_option(
        "-a",
        "--agreement",
        dest="agreement",
        default="kappa",
        help="agreement coefficient to calculate",
    )
    parser.add_option(
        "-e",
        "--exclude",
        dest="exclude",
        action="append",
        default=[],
        help="coder names to exclude (may be specified multiple times)",
    )
    parser.add_option(
        "-i",
        "--include",
        dest="include",
        action="append",
        default=[],
        help="coder names to include, same format as exclude",
    )
    parser.add_option(
        "-f",
        "--file",
        dest="file",
        help="file to read labelings from, each line with three columns: 'labeler item labels'",
    )
    parser.add_option(
        "-v",
        "--verbose",
        dest="verbose",
        default="0",
        help="how much debugging to print on stderr (0-4)",
    )
    parser.add_option(
        "-c",
        "--columnsep",
        dest="columnsep",
        default="\t",
        help="char/string that separates the three columns in the file, defaults to tab",
    )
    parser.add_option(
        "-l",
        "--labelsep",
        dest="labelsep",
        default=",",
        help="char/string that separates labels (if labelers can assign more than one), defaults to comma",
    )
    parser.add_option(
        "-p",
        "--presence",
        dest="presence",
        default=None,
        help="convert each labeling into 1 or 0, based on presence of LABEL",
    )
    parser.add_option(
        "-T",
        "--thorough",
        dest="thorough",
        default=False,
        action="store_true",
        help="calculate agreement for every subset of the annotators",
    )
    (options, remainder) = parser.parse_args()

    if not options.file:
        parser.print_help()
        exit()

    logging.basicConfig(level=50 - 10 * int(options.verbose))

    
    data = []
    with open(options.file) as infile:
        for l in infile:
            toks = l.split(options.columnsep)
            coder, object_, labels = (
                toks[0],
                str(toks[1:-1]),
                frozenset(toks[-1].strip().split(options.labelsep)),
            )
            if (
                (options.include == options.exclude)
                or (len(options.include) > 0 and coder in options.include)
                or (len(options.exclude) > 0 and coder not in options.exclude)
            ):
                data.append((coder, object_, labels))

    if options.presence:
        task = AnnotationTask(
            data, getattr(distance, options.distance)(options.presence)
        )
    else:
        task = AnnotationTask(data, getattr(distance, options.distance))

    if options.thorough:
        pass
    else:
        print(getattr(task, options.agreement)())

    logging.shutdown()