<jupyter_start><jupyter_text>The magic of 2 by 2 tables<jupyter_code># Our usual imports
import numpy as np
import pandas as pd
# Safe settings for Pandas.
pd.set_option('mode.chained_assignment', 'raise')
# HIDDEN
# An extra tweak to make sure we always get the same random numbers.
# Do not use this in your own code; you nearly always want an unpredictable
# stream of random numbers.  Making them predictable in this way only makes
# sense for a very limited range of things, like tutorials and tests.
np.random.seed(1938)<jupyter_output><empty_output><jupyter_text>At the end of the [noble politics](noble_politics) page, you came across anenigma, from analyzing two by two tables of counts.This page shows you another version of that enigma, and gives a fewexplanations for these strange effects.To start with, we make a simple data frame, where each row correspondsto a person.  Each person, and therefore, each row, has two labels.  If youhave read the Noble Politics page, you will see what inspired this example.The first label for each person, called "Party" can be either "Yes", or "No".The second label for each person, called "Respondent", can be either "R" or"B".Here is our data frame from scratch:<jupyter_code># Create a data frame from scratch with some data.
people = pd.DataFrame([["Yes", "R"],
                       ["Yes", "R"],
                       ["Yes","R"],
                       ["No", "R"],
                       ["No", "R"],
                       ["Yes","B"],
                       ["Yes","B"],
                       ["No","B"]],
                       columns = ['Respondent', 'Party'])
# Show the result
people<jupyter_output><empty_output><jupyter_text>We can ask Pandas to make a 2 by 2 table of counts, for these labels.  Eachentry in the table gives the number of times the combination of labels occurs.For example, the cell in the "Yes" row, "R" column gives the number of rows inthe original table that have a "Yes" label for "Party" *and* a "R" label for"Respondent".  Call this the *Yes-with-R* cell.<jupyter_code>two_by_two = pd.crosstab(people['Party'], people['Respondent'])
two_by_two<jupyter_output><empty_output><jupyter_text>You might want to confirm these counts.  You should see that there are in fact3 rows which have both the "Yes" label and the "R" label, and this matches thebottom right entry in the `pd.crosstab` table.We can add the row and column totals to this table, with some fancy Pandascode (unhide the code in this page, or see the notebook version for details):<jupyter_code># Fancy code for adding row / column totals, giving them colors.
from matplotlib.colors import ListedColormap

PALE_GREY = ListedColormap([0.8, 0.8, 0.8])
MID_GREY = ListedColormap([0.6, 0.6, 0.6])

def with_margins(df):
    """ Data frame displayed with row, column totals added, colored.
    """
    # Add row totals to copy of data frame.
    row_margin = df.sum().astype(int).rename('Row totals')
    df_margins = df.append(row_margin)
    # Add column totals.
    df_o = df_margins.assign(**{'Col totals': df_margins.sum(axis='columns')})
    # Set colors for row, column totals.
    styler = df_o.style.background_gradient(subset='Col totals', cmap=PALE_GREY)
    styler = styler.background_gradient(subset=('Row totals',), cmap=PALE_GREY)
    styler = styler.background_gradient(subset=('Row totals', 'Col totals'),
                                                cmap=MID_GREY)
    return styler
# Display the table with colors, using fancy plotting function.
with_margins(two_by_two)<jupyter_output><empty_output><jupyter_text>Now let's say we want to create a random association between the "Party" (Yes/No) column and the "Respondent" (R/B) column.   We can do that by shuffling (permuting) the Yes/No values, and recreating the table, like this:<jupyter_code>parties = people['Party']
respondents = people['Respondent']
permuted_tab_1 = pd.crosstab(np.random.permutation(parties), respondents)
with_margins(permuted_tab_1)<jupyter_output><empty_output><jupyter_text>Note that I could have shuffled the "Respondent" labels instead, or both setsof labels - any of these operations creates what we want - which is a randomassociation between the "Party" and "Respondent" labels.Let's make another few of those random tables:<jupyter_code>permuted_tab_2 = pd.crosstab(np.random.permutation(parties), respondents)
with_margins(permuted_tab_2)
permuted_tab_3 = pd.crosstab(np.random.permutation(parties), respondents)
with_margins(permuted_tab_3)<jupyter_output><empty_output><jupyter_text>You will see that, if the top left value goes up, or down, compared to theoriginal table, then the bottom right value goes up, or down, by the sameamount.  Call the change in the top-right value, from the original `c`.  Theoriginal value for that element was 1, so if the new value is 2, $c = 1$.  Ifthe new value was 0, then $c = -1$.You will also see that the bottom left value goes *down* by the same amount asthe top left value goes up.  If the top left value goes up by $c$, then thebottom left value does down by $c$.All the values in the table appear to be linked, and move in lock-step.Notice too that the column and row totals do not --- and in fact cannot ---change, whatever the association of the R/B and Yes/No labels, for reasons we go into below.This is another version of the enigma you saw at the end of the [NoblePolitics](noble_politics) page.  Why are the table values linked?  Explanation 1: the row and column totals cannot changeThe first explanation for the relationship between the values is --- the totalcounts over the rows and columns are fixed.They are fixed because they depend only on the labels, not on the relationship of the labels.   So, we cannot change the counts over rows or columns by shuffling the order of either or both set of labels.For example, consider the sum of counts over columns.  In our example, thecolumns are for "B" and "R", and the column sums of the counts are the totalnumber of "B" and "R" labels.By shuffling the order of the labels, we can change the number in theB-with-Yes cell, but we can't change the total number of "B" labels, so, ifthe B-with-Yes number increases, the B-with-No number has to decrease by thesame amount.The same argument applies for the row totals.You may now see that, if you give me the value for any one of the fourelements in the table, I can fill in the rest, because they follow from thefact that I know what the rows and columns have to add up to.  Explanation 2: what goes up, must come downHere we think about what happens as we permute the Yes/No labels.  The sameargument applies to permuting the R/B labels, and in fact, to permuting boththe labels.Consider the original table.<jupyter_code>with_margins(two_by_two)<jupyter_output><empty_output><jupyter_text>Now consider permutations in the order of the Yes/No column of the originaltable.  We will just permute the first "Yes", for now.Imagine the first "Yes" gets swapped with the "Yes" in one of the other "R","Yes" rows.  We still end up with the same two by two counts table.Next swap the first "Yes" with the "No" in one of the "R", "No" rows.  Thefirst row is "R", "No" but the other row has now become "R", "Yes", we stillhave the same number of "R", "Yes" and "R", "No", and the counts table doesn'tchange.Now swap the first "Yes" with the "Yes" in one of the "B", "Yes" rows.  Westill have the same number of "R", "Yes" and "B", "Yes" rows, just in adifferent row order, and the counts table doesn't change.Last, we swap the first "Yes" with the "No" in one of the "B", "No" rows. Now,for the first time, the counts table does change, like this:<jupyter_code># The data frame with first "Yes" permuted to last row.
perm_people = pd.DataFrame([["No", "R"],
                            ["Yes", "R"],
                            ["Yes","R"],
                            ["No", "R"],
                            ["No", "R"],
                            ["Yes","B"],
                            ["Yes","B"],
                            ["Yes","B"]],
                            columns = ['Respondent', 'Party'])
perm_people<jupyter_output><empty_output><jupyter_text>Here is the cross-tabulation for that case:<jupyter_code>perm_crosstab = pd.crosstab(perm_people['Party'], perm_people['Respondent'])
with_margins(perm_crosstab)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09059358544758257, 'fraction_numerical': 0.0040689325035902345, 'mean_word_length': 3.708169014084507, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '208609', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 2513, 'n_tokens_neox': 2406, 'n_words': 1087, 'path': 'wild-pandas/two_by_two_tables.ipynb', 'repo_name': 'uob-ds/cfd2021', 'size': '8356'}