<jupyter_start><jupyter_text>Using Audit-AI for Testing Adverse Impact over Time: An Introduction to Cochran-Mantel-Haenzel and Breslow-Day Statistics Contributing data scientists: <NAME> & <NAME>The Audit-AI package provides developers with simple functions that can be used to test for adverse impact in an algorithm. Many of the original functions in this repo were for testing single instances of an algorithm. In application, however, it is often necessary to test for historic trends in a decision-making pipeline. This pipeline could be any number of decisions: the college admissions rate of men vs women over the past decade, the engine failure rate of six major car manufacturers for the past 18 months, or even the hourly number of ad clicks over time in two website formats during AB testing.Here we illustrate the pairwise special case of the Cochran-Mantel-Haenzel test. The CMH test is a generalization of the McNemar Chi-Squared test of Homogenaity. Whereas the McNemar test examines differences over two intervals (usually before and after), the CMH test examines differences over any number of _k_ instances. A Note on Significance Testing for Adverse ImpactBasic researchers often use the CMH test to identify significant differences between groups caused by an experimental condition. In this case, statisticians recommend the application of Yates' correction for continuity. This acts to reduce the test statistics and increase the p-value of tests to correct for false discovery rate at moderately large samples (recommended by some experts to include N's of tens to hundreds of datapoints). Yate's correction is a conservative approach in experimental settings, but NOT in adverse impact monitoring; such an approach would systematically allow marginal cases of bias to go undetected.Users are also advised to take practical significance into account. Statistical significance may be achieved at sufficiently large sample sizes despite trivially small effect sizes. Users should consult the industry, academic and regulatory norms for practical significance in their use-case. Example: McDonald and Siebenaller (1989) Example taken from the Handbook of Biological Statistics by <NAME> (http://www.biostathandbook.com/cmh.html). From the text:    "McDonald and Siebenaller (1989) surveyed allele frequencies at the Lap    locus in the mussel Mytilus trossulus on the Oregon coast. At four    estuaries, we collected mussels from inside the estuary and from a marine    habitat outside the estuary. There were three common alleles and a couple    of rare alleles; based on previous results, the biologically interesting    question was whether the Lap94 allele was less common inside estuaries,    so we pooled all the other alleles into a "non-94" class."    "There are three nominal variables: allele (94 or non-94), habitat    (marine or estuarine), and area (Tillamook, Yaquina, Alsea, or Umpqua).    The null hypothesis is that at each area, there is no difference in the    proportion of Lap94 alleles between the marine and estuarine habitats."<jupyter_code>import numpy as np
import pandas as pd
import auditai
tillamook = pd.DataFrame({'Marine':[56,40],'Estuarine':[69,77]}, index=['94','non-94'])
tillamook
yaquina = pd.DataFrame({'Marine':[61,57],'Estuarine':[257,301]}, index=['94','non-94'])
alsea = pd.DataFrame({'Marine':[73,71],'Estuarine':[65,79]}, index=['94','non-94'])
umpqua = pd.DataFrame({'Marine':[71,55],'Estuarine':[48,48]}, index=['94','non-94'])
dfs = [tillamook,yaquina,alsea,umpqua]

dfs
# import the CMH test from auditai
from auditai.stats import test_cmh_bd
# pass_col is reduntant here, as either case could be considered "passing" a priori
# setting `pass_col` to False compares collumns in order (Estuarine compared to Marine)
# Note that all statistics are identical, except `r`, the common odds ratio, 
# which will be inverted
r, cmh, pcmh, bd, pbd = test_cmh_bd(dfs=dfs, pass_col=False) 
print ("common odds ratio: {}".format(r))
print ("CMH Chi-Squared Statistic: {}".format(cmh))
print ("CMH p-value: {}".format(pcmh))
print ("Breslow-Day Chi-Squared Statistic: {}".format(bd))
print ("Breslow-Day p-value: {}".format(pbd))

test_cmh_bd(dfs=dfs, pass_col='Estuarine')<jupyter_output><empty_output><jupyter_text>Explaining the FunctionsThe `test_cmh_bd` function is a wrapper for three statistical analyses. Odds ratioThe `multi_odds_ratio` function computes the common odds ratio from average of classifications across multiple samples. The odds ratio of the total  _will_ diverge from the odds ratio at each sampling interval.<jupyter_code>from auditai.stats import multi_odds_ratio

print ('odds ratio at each sampling interval')
for df in dfs:
    print (multi_odds_ratio(df))

print ('\nodds ratio for the total sample')
print (multi_odds_ratio(dfs))<jupyter_output>odds ratio at each sampling interval
0.640074211502783
0.7978323620717825
0.8002427605340732
0.7746478873239436

odds ratio for the total sample
0.7590219991052517<jupyter_text>Interested parties will note that the common odds ratio of the total is not just the harmonic mean of the sample odds ratios.<jupyter_code>np.mean([1.56231884058,1.25339613626,1.24962080173,1.29090909091])<jupyter_output><empty_output><jupyter_text>Nor is the total common odds ratio just the odds ratio of the sum of all sample measurements.<jupyter_code>dfTOT = dfs[0] + dfs[1] + dfs[2] + dfs[3]
print (multi_odds_ratio(dfTOT))<jupyter_output>0.742741170668791<jupyter_text>Rather, the total odds ratio is an approximation of the true, unknown odds ratio, taken from the ratio of "pass" and "fail" observations of both categories. Note that this method works even in the current case, where there is not a true pass or fail category.<jupyter_code>from auditai.utils.cmh import parse_matrix, extract_data
from functools import partial
r_num = []
r_den = []
for df in dfs:
    pass0, fail0, pass1, fail1, total = parse_matrix(df)
    r_num.append((pass0*fail1)/(total))
    r_den.append((fail0*pass1)/(total))
print (sum(r_num), sum(r_den))
print (float(sum(r_num))/float(sum(r_den)))<jupyter_output>60.99127446832867 80.35508132863902
0.7590219991052517<jupyter_text>CMH TestThe `cmh_test` function computes the Cochran-Mantel-Haenszel chi-squared statistic and corresponding p-value. The CMH test statistic grows as the observed values from one group deviates from the pooled expected value of both groups.From McDonald http://www.biostathandbook.com/cmh.html :    "The numerator contains the absolute value of the difference between the observed value in one cell (a) and the expected value under the null hypothesis, (a+b)(a+c)/n, so the numerator is the squared sum of deviations between the observed and expected values. It doesn't matter how you arrange the 2×2 tables, any of the four values can be used as a. You subtract the 0.5 as a continuity correction. The denominator contains an estimate of the variance of the squared differences."    "The test statistic, χ2MH, gets bigger as the differences between the observed and expected values get larger, or as the variance gets smaller (primarily due to the sample size getting bigger). It is chi-square distributed with one degree of freedom."<jupyter_code>from auditai.stats import cmh_test
cmh_test(dfs)<jupyter_output><empty_output><jupyter_text>The CMH test statistic is sensitive to sample size at each interval<jupyter_code>from copy import deepcopy
dfs2 = deepcopy(dfs)
dfs2[0] = dfs[0] * 10
cmh_test(dfs)
cmh_test(dfs2)<jupyter_output><empty_output><jupyter_text>...and is primarially suited to identify trends over time, which are magnified at large sample sizes.<jupyter_code>dfs2 = deepcopy(dfs)
for i,d in enumerate(dfs2):
    dfs2[i] = d * 10
dfs2
cmh_test(dfs2)<jupyter_output><empty_output><jupyter_text>Breslow-Day TestThe `bres_day` function computes the Breslow-Day test of homogeneous association for a 2 x 2 x k table. For example, given three factors, A, B, and C, the Breslow-Day test would measure wheher pairwise effects (AB, AC, BC) have identical odds ratios.Here, we see that the original set of sample data, `dfs`, have relatively consistent directionality of odds ratios of differences between regions and alelle groups. A modified set of data, where the odds ratio of one of the groups is greater and in the opposite direction than odds ratios of other samples.<jupyter_code>dfs2 = deepcopy(dfs)
dfs2[0].iloc[0,0] = dfs2[0].iloc[0,0] * 10
from auditai.stats import bres_day
r = multi_odds_ratio(dfs)
part_bd = partial(bres_day, r=r)
# sum of Breslow-Day chi-square statistics
bd = pd.DataFrame(map(part_bd, dfs))[0].sum()

r2 = multi_odds_ratio(dfs2)
part_bd2 = partial(bres_day, r=r2)
bd2 = pd.DataFrame(map(part_bd, dfs2))[0].sum()

print(bd,bd2)
r2 = multi_odds_ratio(dfs2)
for df in dfs2:
    r = multi_odds_ratio(df)
    bd, pbd = bres_day(df,r2)
    print (r,bd,pbd)<jupyter_output>0.06400742115027828 78.79661720061715 0.0
0.7978323620717825 8.565769483918876 0.003425421688411645
0.8002427605340732 6.319607038889988 0.01194100883813387
0.7746478873239436 4.2598761264502505 0.039022766893019756
{'fraction_non_alphanumeric': 0.07700757991870812, 'fraction_numerical': 0.05668460946940569, 'mean_word_length': 5.611474219317357, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '304907', 'license': 'MIT', 'n_tokens_mistral': 3099, 'n_tokens_neox': 2724, 'n_words': 1112, 'path': 'examples/CMH_test_example.ipynb', 'repo_name': 'vishalbelsare/audit-ai', 'size': '9128'}