<jupyter_start><jupyter_text>Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your name and collaborators below:<jupyter_code>NAME = ""
COLLABORATORS = ""<jupyter_output><empty_output><jupyter_text>Neuro Data Analysis in Python: Problem Set 4This is the fourth problem set. It has 3 problems, worth a total of 41 points. It is due before class (i.e. by 10:59 AM) on 10/30/2020. For late policy please see [the syllabus](https://github.com/alexhuth/ndap-fa2020/blob/master/README.mdlate-homework--extension-policy). Partial credit will be awarded for partially correct solutions. Homework submissionWhen you've finished, rename the notebook file to `ndap-problem_set_4-YOUREID.ipynb`. For example, if your EID is `ab12345`, you should call it `ndap-problem_set_4-ab12345.ipynb`. Then upload your completed problem set to canvas.<jupyter_code>%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import numpy as np
from matplotlib import pyplot as plt
from scipy import stats<jupyter_output><empty_output><jupyter_text>--- Problem 1. (9 pts)Solve each part.<jupyter_code># run this cell to create some data
my_data_1 = np.random.randn(100)
my_data_2 = my_data_1 + np.random.randn(100)
# compute the mean of my_data_1 (1 pt)
### YOUR CODE HERE ###
# compute the median of my_data_1 (1 pt)
### YOUR CODE HERE ###
# compute the variance (not sample variance) of my_data_1 (1 pt)
### YOUR CODE HERE ###
# compute the SAMPLE variance of my_data_1 (1 pt)
### YOUR CODE HERE ###
# compute the standard deviation (not sample standard deviation) of my_data_1 (1 pt)
### YOUR CODE HERE ###
# compute the SAMPLE standard deviation of my_data_1 (1 pt)
### YOUR CODE HERE ###
# compute the correlation of my_data_1 and my_data_2 (1 pt)
# (this should only be a single number, not a matrix!)
### YOUR CODE HERE ###
# compute the covariance of my_data_1 and my_data_2 (1 pt)
# (this should also just be a single number)
### YOUR CODE HERE ###
# compute the 10th and 90th percentiles of my_data_1 (1 pt)
### YOUR CODE HERE ###<jupyter_output><empty_output><jupyter_text>*** Problem 2. (16 pts)Suppose that we have run an experiment to see whether a new training strategy changes subjects' performance on a task. We first test 200 subjects on the task without training them, giving us the scores `d1`. Then we get 200 more subjects, train them using the new strategy, and then test how well they perform the task, giving the scores `d2`. We want to know whether the performance is different between these two groups (but we don't care which direction the difference lies).In this problem you'll test whether these two datasets have significantly different means using a few different techniques.<jupyter_code># Load the data
datafile = np.load('ps4_dataset_2.npz')
d1 = datafile['data1']
d2 = datafile['data2']

# Let's see the size of each array
print(d1.shape)
print(d2.shape)

# and print the mean of each
print('d1 mean:', d1.mean())
print('d2 mean:', d2.mean())<jupyter_output><empty_output><jupyter_text>(a) Use a t-test to see whether the means are significantly different (2 pts)Use the function `ttest_ind` to test whether the means of the two dataset are significantly different. Print the p-value. Note that this is a 2-sided test because it is testing whether the two are _different_ (i.e. either d1's mean is greater than d2, or vice versa).Print the p-value.<jupyter_code>### YOUR CODE HERE ###<jupyter_output><empty_output><jupyter_text>(b) Use a bootstrap test to see whether the means are different (7 pts)Next, design and use a bootstrap test to tell whether the two datasets have significantly different means. This is a little different than the bootstrap procedures we used previously in class, because here we will use the bootstrap to generate a distribution of the null hypothesis. See [here](https://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29Bootstrap_hypothesis_testing) for a different description of this procedure.Here's the procedure you should use:1. Create a new dataset `d1_2` by subtracting the mean of d1 from d1, and then adding the mean of the combined sample (i.e. the concatenation of d1 and d2)2. Similarly create a new dataset `d2_2` by subtracting the mean of d2 from d2, and then adding the combined mean3. Draw bootstrap samples from `d1_2` and `d2_2`, and compute the difference of their means (the function `np.random.choice` might be useful here)4. Repeat step 3 many (10,000) times5. Compute the p-value of the difference between d1 and d2 by comparing the actual difference in means between d1 and d2 to the bootstrap samplesThe bootstrap p-value that you get at the end should be about the same as what you got for the t-test.<jupyter_code># create new datasets d1_2 and d2_2
d1_2 = ### YOUR CODE HERE ###
d2_2 = ### YOUR CODE HERE ###

# create array to hold bootstrap mean differences
nbootstraps = 10000
bs_mean_diffs = ### YOUR CODE HERE ###

# take bootstrap samples many times
for ii in range(nbootstraps):
    # choose which indices will be used from d1_2 and d2_2
    inds1 = ### YOUR CODE HERE ###
    inds2 = ### YOUR CODE HERE ###
    
    # create your bootstrap samples
    bs_d1 = ### YOUR CODE HERE ###
    bs_d2 = ### YOUR CODE HERE
    
    # measure their difference and store it
    ### YOUR CODE HERE ###

# take the absolute value of each bootstrap difference, and find the fraction that are 
# larger than the absolute mean difference between d1 and d2. this is the bootstrap p-value
bs_pval = ### YOUR CODE HERE ###

print('bootstrap p-value:', bs_pval)<jupyter_output><empty_output><jupyter_text>(c) Use a permutation test to see whether the means are different (7 pts)Next, design and use a permutation test to tell whether the two datasets have significantly different means. That'll work like this:1. Lump both datasets together into one using `np.hstack`.2. Randomly permute (shuffle) the lumped dataset using `np.random.permutation`3. Split the shuffled data into two datasets the same size as d1 and d24. Compute the difference between the means of the new shuffled datasets5. Repeat steps 2-4 many (10,000) times6. Compare the actual difference to the permutation samples to get the p-valueThe permutation p-value you get at the end should be about the same as what you got for the t-test and bootstrap test.<jupyter_code># lump both datasets together
d_lump = ### YOUR CODE HERE ###

# create array to hold permutation differences
npermutations = 10000
p_mean_diffs = ### YOUR CODE HERE ###

# permute & compute many times
for ii in range(npermutations):
    # permute d_lump
    perm_d_lump = ### YOUR CODE HERE ###
    
    # split it into two parts and find the difference of their means
    perm_d1 = ### YOUR CODE HERE ###
    perm_d2 = ### YOUR CODE HERE ###
    
    # compute the difference of their means and store it
    ### YOUR CODE HERE ###


# take the absolute value of each permutation difference, and find the fraction that are 
# larger than the absolute mean difference between d1 and d2. this is the permutation p-value
p_pval = ### YOUR CODE HERE ###

print('permutation p-value:', p_pval)<jupyter_output><empty_output><jupyter_text>--- Problem 3. (16 pts)Now suppose that we've realized our initial experimental design was a nightmare. (400 subjects, who has the time!) So for the next experiment (and new training strategy) we've gone with a different design. This time, we test each of the 200 subjects (as before, yielding the dataset `e1`), _then_ train them using our new training strategy, then test each subject _again_, yielding the dataset `e2`.This time the datasets are _paired_, meaning that `e1[0]` and `e2[0]` are from the same subject. We have to account for this in our analyses, because what we really care about is how much each subject _changed_.<jupyter_code># Load the data
datafile = np.load('ps4_dataset_3.npz')
e1 = datafile['data1']
e2 = datafile['data2']

# Let's see the size of each array
print(e1.shape)
print(e2.shape)

# and print the mean of each
print('e1 mean:', e1.mean())
print('e2 mean:', e2.mean())<jupyter_output><empty_output><jupyter_text>(a) Use a t-test and paired t-test to compare means (2 pts)First, use a normal t-test (as before) to compare the means of `e1` and `e2`. Then, use a paired t-test (`ttest_rel`). Print the p-values for both.<jupyter_code>### YOUR CODE HERE ###<jupyter_output><empty_output><jupyter_text>(b) Use a paired bootstrap test to compare the means (7 pts)Design and run a paired bootstrap test to compare the means of `e1` and `e2`. This should look almost identical to the simple bootstrap test above, but with one critical difference. Think hard about what the bootstrap samples represent, and how to make it paired. As before, use 10,000 bootstrap samples.Print the bootstrap p-value at the end.<jupyter_code>### YOUR CODE HERE ###<jupyter_output><empty_output><jupyter_text>(c) Use a paired permutation test to compare the means (7 pts)Design and run a paired permutation test to compare the means of `e1` and `e2`. Unlike for the bootstrap, the paired permutation test is actually quite different from the normal permutation test. Remember that you need to keep the pairs of datapoints together. Think hard about what the null hypothesis (i.e. that the training had no effect) would mean when you do permutation. As before, do 10,000 permutations.Print the permutation p-value at the end.<jupyter_code>### YOUR CODE HERE ###<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08225241918879968, 'fraction_numerical': 0.02285361334156887, 'mean_word_length': 4.707990599294948, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '133883', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2999, 'n_tokens_neox': 2793, 'n_words': 1349, 'path': 'problem_sets/ps4/ndap-problem_set_4.ipynb', 'repo_name': 'alexhuth/ndap-fa2020', 'size': '9714'}