<jupyter_start><jupyter_text>Comparing two groups with permutation testingWe return to the Brexit age data from [Where are the Brexiteers?](https://matthew-brett.github.io/les-pilot/brexit_problem.html).Here’s the data from the processing we did in [Where are the Brexiteers?](https://matthew-brett.github.io/les-pilot/brexit_analysis.html):* [remain_leave.csv](https://matthew-brett.github.io/les-pilot/_downloads/remain_leave.csv).Download the data file from the link above if you haven’t already got it inyour Notebook directory.<jupyter_code>import pandas as pd
remain_leave = pd.read_csv('remain_leave.csv')
remainers = remain_leave[remain_leave['brexit'] == 1]
brexiteers = remain_leave[remain_leave['brexit'] == 2]
# Confirm our proportions haven't changed
len(brexiteers) / (len(brexiteers) + len(remainers))<jupyter_output><empty_output><jupyter_text>For convenience we get our ages scores into lists for each group:<jupyter_code># We make a list from the Pandas column with the "list" function
brexit_ages = list(brexiteers['age'])
remain_ages = list(remainers['age'])
# Check our proportions again
len(brexit_ages) / (len(brexit_ages) + len(remain_ages))
brexit_ages[:10]
remain_ages[:10]<jupyter_output><empty_output><jupyter_text>Let’s put up the histograms of these two groups again:<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
plt.hist(brexit_ages)
plt.hist(remain_ages)<jupyter_output><empty_output><jupyter_text>The remainers look as though they may be a bit younger on average.Let’s look at the mean age for the two groups.The *mean* of the values in the list is defined as the sum divided bythe number of items.  Here is the mean age of the Brexit Leave voters:<jupyter_code>sum(brexit_ages) / len(brexit_ages)<jupyter_output><empty_output><jupyter_text>We can define a function to calculate the mean:<jupyter_code>def mean(some_list):
    # Calculates the mean of the values in `some_list`
    return sum(some_list) / len(some_list)<jupyter_output><empty_output><jupyter_text>Now we can get the mean with:<jupyter_code>mean(brexit_ages)<jupyter_output><empty_output><jupyter_text>The mean is lower for the Remain group:<jupyter_code>mean(remain_ages)<jupyter_output><empty_output><jupyter_text>The difference between the means is:<jupyter_code>observed_difference = mean(brexit_ages) - mean(remain_ages)
observed_difference<jupyter_output><empty_output><jupyter_text>We see that the means of the two groups are different - but can we beconfident that this difference did not come about by chance?What do we mean by chance?  Now we have to define our null hypothesis.We hypothesize that there is in fact no difference between the ages of the twogroups.  We propose that the difference between the means we see couldreasonably occur if we did the following procedure:* sample 541 + 774 = 1315 people from this same underlying  doesn’t-matter-what-you-voted population;* arbitrarily split this sample into a group of 541 and a group of 774, call  these `group_A` and `group_B`;* calculate the mean age for `group_A` and subtract the mean age of  `group_B`;Call this procedure - one trial.  To test our null hypothesis, we would liketo repeat this trial many times, recording the difference in means for eachtrial. By doing this we could build up a distribution of the kind ofdifferences we would expect by chance - the sampling distribution.But - we don’t have any more data - so we don’t have many new groups of 541 +774 = 1315 to sample.  But it turns out, we can use the data we have to buildthe distribution we need.Let’s start with the null hypothesis - that there is no difference in the agesof the Leave and Remain groups. If that hypothesis is true, then all theages in `brexit_ages` and in `remain_ages` can be considered asbeing samples from the same underlying group.To express this, I’m going to pool all the weights into one big group, likethis:<jupyter_code># The + below appends the second list to the first
all_ages = brexit_ages + remain_ages
len(all_ages)<jupyter_output><empty_output><jupyter_text>In this new pooled list, the first 541 ages are from the `brexit_ages` list,and the rest are from the `remain_ages` list.Now we have the new pooled list, we can do something similar to taking the new`group_A` and `group_B` groups we imagined above.  That is, we can*shuffle* the combined group to a random order, and split this shuffledcombined group into a group of 541 and a group of 774.  We get the differencein means of these two groups, and store it.  We keep shuffling, to create more`group_A` and `group_B` groups, and more differences in means. Thegenerated distribution of the mean differences is the distribution we expecton the null hypothesis, that there is no real difference between the twogroups.  We can see where `observed_difference` lies in this generateddistribution, and this gives us a measure of whether the`observed_difference` is likely on the null hypothesis.Python’s `random.shuffle` function can do the shuffle for us (see:[Shuffling list element order](https://matthew-brett.github.io/les-pilot/more_on_lists.htmllist-shuffle)):<jupyter_code>import random<jupyter_output><empty_output><jupyter_text>Before shuffling, the first 541 age values are all for Leave voters.  Here arethe first 10:<jupyter_code># The first 10 ages before shuffling (all brexit)
all_ages[:10]<jupyter_output><empty_output><jupyter_text>Here’s a random shuffle of the combined list of ages:<jupyter_code>random.shuffle(all_ages)
# The first 10 ages of the shuffled list, mixed brexit and remain.
all_ages[:10]<jupyter_output><empty_output><jupyter_text>To express all this, I’m going to make a function that takes the two lists,joins them, shuffles, them and returns two new lists, that have a randomassortment of values from the original list:<jupyter_code># Just in case you haven't imported this already
import random
def join_shuffle_split(first_list, second_list):
    # Pool the lists, shuffle, split into lists of original length
    first_len = len(first_list)
    pooled = first_list + second_list
    random.shuffle(pooled)
    both = [pooled[:first_len], pooled[first_len:]]
    return both<jupyter_output><empty_output><jupyter_text>Let’s check that is working as expected:<jupyter_code># Make a couple of lists
a_list = [0, 0, 0]
b_list = [99, 99, 99, 99]
# Call the function to shuffle the lists
join_shuffle_split(a_list, b_list)
join_shuffle_split(a_list, b_list)<jupyter_output><empty_output><jupyter_text>Now for our permutation test.  We’ve assumed the null hypothesis.  We join thelists, and shuffle the combined group.  We’ll call the first 541 values`group_A` and the last 774 values `group_B`.<jupyter_code>fake_lists = join_shuffle_split(brexit_ages, remain_ages)
group_A = fake_lists[0]
group_B = fake_lists[1]<jupyter_output><empty_output><jupyter_text>After the shuffling, the `group_A` group is a random mix of the`brexit_ages` and `remain_ages` values, as is the `group_B` group.Here is a function that takes the original lists, makes `group_A` and`group_B`, and returns the difference in means.<jupyter_code>def difference_in_means(first_list, second_list):
    """ Shuffle values across lists, calculate difference in means.
    """
    fake_lists = join_shuffle_split(first_list, second_list)
    group_A = fake_lists[0]
    group_B = fake_lists[1]
    return mean(group_B) - mean(group_A)<jupyter_output><empty_output><jupyter_text>Let’s get the difference in means for these new groups, generated by theshuffle:<jupyter_code>difference_in_means(brexit_ages, remain_ages)<jupyter_output><empty_output><jupyter_text>That difference from the shuffled groups looks a lot less than the differencewe originally found:<jupyter_code>observed_difference<jupyter_output><empty_output><jupyter_text>One difference is not enough.  We need more shuffled mean differences to seewhether `observed_difference` is really unusual compared to the range ofpermuted group differences.  Here we run the shuffle procedure 10000 times, toget a large range of values:<jupyter_code>n_repeats = 10000
shuffled_differences = []  # An empty list to store the differences
for i in range(n_repeats):
    fake_difference = difference_in_means(brexit_ages, remain_ages)
    # Collect the new mean by adding to the end of the list
    shuffled_differences.append(fake_difference)<jupyter_output><empty_output><jupyter_text>This gives us 10000 differences from groups that are compatible with our nullhypothesis.   We can now ask whether `observed_difference` is unusuallylarge compared to the distribution of these 10000 differences.<jupyter_code>%matplotlib inline
# The plotting package
import matplotlib.pyplot as plt
plt.hist(shuffled_differences)<jupyter_output><empty_output><jupyter_text>The maximum of these differences is:<jupyter_code>max(shuffled_differences)<jupyter_output><empty_output><jupyter_text>Remember our `observed_difference`?<jupyter_code>observed_difference<jupyter_output><empty_output><jupyter_text>So - how many of the `shuffled_differences` are greater than or equal to the`observed_difference`?<jupyter_code>n_greater_equal = 0
for i in range(n_repeats):
    if shuffled_differences[i] >= observed_difference:
        n_greater_equal = n_greater_equal + 1
n_greater_equal
2 / 10000<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07344875793096031, 'fraction_numerical': 0.011076459834390795, 'mean_word_length': 6.259953161592506, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 94, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '410267', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3011, 'n_tokens_neox': 2828, 'n_words': 1016, 'path': 'brexit_ages_in_class.ipynb', 'repo_name': 'matthew-brett/eef-pilot1', 'size': '9299'}