<jupyter_start><jupyter_text>Inter-rater agreement Kappasa.k.a. inter-rater reliability or concordance> In statistics, inter-rater reliability, inter-rater agreement, or concordance is the degree of agreement among raters. It gives a score of how much homogeneity, or consensus, there is in the ratings given by judges.The Kappas covered here are most appropriate for **"nominal"** data. The natural ordering in the data (if any exists) is ignored by these methods. If you're going to use these metrics make sure you're aware of the [limitations](https://www.wikiwand.com/en/Cohen%27s_kappa/Limitations).  Cohen's Kappa \begin{equation*}\kappa =  1 - \frac{1 - p_o}{1 - p_e}\end{equation*}There's two parts to this:1. Calculate observed agreement2. Calculate agreement by chanceLet's say we're dealing with "yes" and "no" answers and 2 raters. Here are the ratings:<jupyter_code>rater1 = ['yes', 'no', 'yes', 'yes', 'yes', 'yes', 'no', 'yes', 'yes']
rater2 = ['yes', 'no', 'no', 'yes', 'yes', 'yes', 'yes', 'yes', 'yes']<jupyter_output><empty_output><jupyter_text>Turning these ratings into a confusion matrix:|           |Rater 2 yes|Rater 2 no||-----------|-----------|----------||Rater 1 yes|6          |1         ||Rater 2 no |1          |1         |```Observed agreement = (6 + 1) / 10 = 0.7Chance agreement   = probability of randomly saying yes (P_yes) + probability of randomly saying no (P_no)P_yes              = (6 + 1) / 10 * (6 + 1) / 10 = 0.49P_no               = (1 + 1) / 10 * (1 + 1) / 10 = 0.04Chance agreement   = 0.49 + 0.04 = 0.53```Since the observed agreement is larger than chance agreement we'll get a positive Kappa.```kappa = 1 - (1 - 0.7) / (1 - 0.53) = 0.36``` Or just use `sklearn`'s implementation<jupyter_code>from sklearn.metrics import cohen_kappa_score
cohen_kappa_score(rater1, rater2)<jupyter_output><empty_output><jupyter_text>Interpretation of Kappa  Different cases  Less than chance agreement<jupyter_code>rater1 = ['no', 'no', 'no', 'no', 'no', 'yes', 'no', 'no', 'no', 'no']
rater2 = ['yes', 'no', 'no', 'yes', 'yes', 'no', 'yes', 'yes', 'yes', 'yes']
cohen_kappa_score(rater1, rater2)<jupyter_output><empty_output><jupyter_text>If all the ratings are the same and oppositeThis case reliably produces a `kappa` of 0<jupyter_code>rater1 = ['yes'] * 10
rater2 = ['no'] * 10
cohen_kappa_score(rater1, rater2)<jupyter_output><empty_output><jupyter_text>Random ratingsFor random ratings `Kappa` follows a normal distribution with a mean of about zero.As the number of ratings increases there's less variability in the value of Kappa in the distribution.<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
np.random.seed(0)


for i in [10, 100, 1000]:
    print('{} random ratings for each rater (random sample of 1,000 inter-rater Kappa calculations)'.format(i))
    kappas = [cohen_kappa_score(np.random.choice(['yes', 'no'], i),
                       np.random.choice(['yes', 'no'], i)) for _ in range(1000)]
    pd.Series(kappas).hist();
    plt.xlim([-1, 1])
    plt.show();
    print('\n')<jupyter_output>10 random ratings for each rater (random sample of 1,000 inter-rater Kappa calculations)<jupyter_text>You can find more details [here](https://www.wikiwand.com/en/Cohen%27s_kappa)**Note that Cohen's Kappa only applied to 2 raters rating the exact same items.**  FleissExtends Cohen's Kappa to more than 2 raters. Interpretation> It can be interpreted as expressing the extent to which the observed amount of agreement among raters exceeds what would be expected if all raters made their ratings completely randomly.The raters can rate different items whereas for Cohen's they need to rate the exact same items> Fleiss' kappa specifically allows that although there are a fixed number of raters (e.g., three), different items may be rated by different individuals\begin{equation*}\kappa =  \frac{\bar p - \bar p_e}{1-\bar p_e}\end{equation*} For example let's say we have 10 raters, each doing a "yes" or "no" rating on 5 items:n_ij | yes | no | P_i--- | --- | --- | ---1 | 10 | 0 | 12 | 8 | 2 | 0.643 | 9 | 1 | 0.84 | 0 | 10 | 15 | 7 | 3 | 0.53**Total** | **34** | **16** | p_j | 0.68 | 0.32 |For example the first row (P_1):```P_1 = (10 ** 2 + 0 ** 2 - 10) / (10 * 9) = 1```And the first columns (p_1):```p_1 = 34 / (5 * 10) = 0.68```Go through the worked example [here](https://www.wikiwand.com/en/Fleiss'_kappa/Worked_example) if this is not clear.Now you can calculate Kappa:```P_bar = (1 / 5) * (1 + 0.64 + 0.8 + 1 + 0.53) = 0.794P_bar_e = 0.68 ** 2 + 0.32 ** 2 = 0.5648```At this point we have everything we need and `kappa` is calculated just as we calculated Cohen's:```kappa = (0.794 - 0.5648) / (1 - 0.5648) = 0.53```<jupyter_code># Adapted from https://gist.github.com/ShinNoNoir/4749548
def fleiss_kappa(ratings, n):
    '''
    Computes the Fleiss' kappa measure for assessing the reliability of 
    agreement between a fixed number n of raters when assigning categorical
    ratings to a number of items.
    
    Args:
        ratings: a list of (item, category)-ratings
        n: number of raters
        k: number of categories
    Returns:
        the Fleiss' kappa score
    
    See also:
        http://en.wikipedia.org/wiki/Fleiss'_kappa
    '''
    items = set()
    categories = set()
    n_ij = {}
    
    for i, c in ratings:
        items.add(i)
        categories.add(c)
        n_ij[(i,c)] = n_ij.get((i,c), 0) + 1
    
    N = len(items)
    
    p_j = dict(((c, sum(n_ij.get((i, c), 0) for i in items) / (1.0 * n * N)) for c in categories))
    P_i = dict(((i, (sum(n_ij.get((i, c), 0) ** 2 for c in categories) - n) / (n * (n - 1.0))) for i in items))

    P_bar = sum(P_i.values()) / (1.0 * N)
    P_e_bar = sum(value ** 2 for value in p_j.values())
    
    kappa = (P_bar - P_e_bar) / (1 - P_e_bar)
    
    return kappa
ratings = [(1, 'yes')] * 10 + [(1, 'no')] * 0  + \
[(2, 'yes')] * 8  + [(2, 'no')] * 2  + \
[(3, 'yes')] * 9  + [(3, 'no')] * 1  + \
[(4, 'yes')] * 0  + [(4, 'no')] * 10 + \
[(5, 'yes')] * 7  + [(5, 'no')] * 3
fleiss_kappa(ratings, 10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15493877551020407, 'fraction_numerical': 0.04342857142857143, 'mean_word_length': 3.385110952040086, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 27, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '328182', 'license': 'MIT', 'n_tokens_mistral': 2317, 'n_tokens_neox': 2098, 'n_words': 729, 'path': 'statistics/Inter-rater agreement kappas.ipynb', 'repo_name': 'ashishpatel26/learning', 'size': '6125'}