Statistical evaluation of panel repeatability in Check All That Apply questions
Abstract Methodologies for evaluating panel repeatability in Check All That Apply (CATA) questions are reviewed and developed. First, the limitations with using McNemar's test as suggested elsewhere for the evaluation of repeatability are demonstrated through simple examples. Alternative approaches are then suggested and discussed. These include the binomial test, Gwet's AC1 statistic, and Pearson's kh2 goodness of fit test. These methodologies are applied to previously published orange juice data. The advantages of using the binomial test or the Pearson's kh2 goodness of fit test are related to their accessibility in most statistical software packages. The advantages of using the Gwet's AC1 statistic or the Pearson's kh2 goodness of fit test are related to the fact that tests are easily generalized to more than 2 replications. Pearson's kh2 goodness of fit test is widely available in statistical software and generalizable to more than 2 replications, but this test is sensitive to very low expected frequencies. For this reason we suggest using Gwet's AC1 statistic, which does not share this limitation.