<jupyter_start><jupyter_text>Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE". --- As a reminder, one of the prerequisites for this course if programming experience, especially in Python. If you do not have experience in Python specifically, we strongly recommend you go through the Codecademy Python course as soon as possible to brush up on the basics of Python. Before going through this notebook, you may want to take a quick look at [7 - Debugging.ipynb](7 - Debugging.ipynb) if you haven't already for some tips on debugging your code when you get stuck. Sometimes, there are more advanced operations we want to do with NumPy arrays. For example, if we had an array of values and wanted to set all negative values to zero, how would we do this? The answer is called *fancy indexing*, and be done two ways: boolean indexing, and array indexing.<jupyter_code>import numpy as np<jupyter_output><empty_output><jupyter_text>Boolean indexing The idea behind boolean indexing is that for each element of the array, we know whether we want to select it or not. A *boolean array* is an array of the same shape as our original array which contains only True and False values. The location of the True values in our boolean array indicate the location of the element in our original array that we want to select, while the location of the False values correspond to those elements in our original array that we _don't_ want to select.Let's consider our experiment data again:<jupyter_code>data = np.load("data/experiment_data.npy")
data<jupyter_output><empty_output><jupyter_text>Recall that these are reaction times. It is typically accepted that really low reaction times -- such as less than 100 milliseconds -- are too fast for people to have actually seen and processed the stimulus. Let's see if there are any reaction times less than 100 milliseconds in our data.To pull out just the elements less than 100 milliseconds, we need two steps. First, we use boolean comparisons to check which are less than 100ms:<jupyter_code>too_fast = data < 100
too_fast<jupyter_output><empty_output><jupyter_text>Then, using this `too_fast` array, we can index *back into* the original array, and see that there are indeed some trials which were abnormally fast:<jupyter_code>data[too_fast]<jupyter_output><empty_output><jupyter_text>What this is doing is essentially saying: for every element in `too_fast` that is `True`, give me the corresponding element in `arr`. Bcause this is a boolean array, we can also negate it, and pull out all the elements that we consider to be valid reaction times:<jupyter_code>data[~too_fast]<jupyter_output><empty_output><jupyter_text>Not only does this *give* you the elements, but modifying those elements will modify the original array, too. In this case, we will set our "too fast" elements to have a value of "not a number", or `NaN`:<jupyter_code>data[too_fast] = np.nan
data<jupyter_output><empty_output><jupyter_text>Now, if we try to find which elements are less than 100 milliseconds, we will not find any:<jupyter_code>data[data < 100]<jupyter_output><empty_output><jupyter_text>Note: You may see a RuntimeWarning when you run the above cell, saying that an "invalid value" was encountered. Sometimes, it is possible for NaNs to appear in an array without your knowledge: for example, if you multiply infinity (np.inf) by zero. So, NumPy is warning us that it has encountered NaNs (the "invalid value") in case we weren't aware. We knew there were NaNs because we put them there, so in this scenario we can safely ignore the warning. However, if you encounter a warning like this in the future and you weren't expecting it, make sure you investigate the source of the warning!  Exercise: Threshold Write a function, threshold, which takes an array and returns a new array with values thresholded by the mean of the array.<jupyter_code>def threshold(arr):
    """Computes the mean of the given array, and returns a new array which
    is 1 where values in the original array are greater than the mean, 0 where
    they are equal to the mean, and -1 where they are less than the mean.

    Remember that if you want to create a copy of an array, you need to use
    `arr.copy()`.
    
    Hint: your solution should use boolean indexing, and can be done in six
    lines of code (including the return statement).
    
    Parameters
    ----------
    arr : numpy.ndarray

    Returns
    -------
    new_arr : thresholded version of `arr`
    
    """
    # YOUR CODE HERE
    raise NotImplementedError()
# add your own test cases in this cell!

"""Try a few obvious threshold cases."""
from numpy.testing import assert_array_equal
assert_array_equal(threshold(np.array([1, 1, 1, 1])), np.array([0, 0, 0, 0]))
assert_array_equal(threshold(np.array([1, 0, 1, 0])), np.array([1, -1, 1, -1]))
assert_array_equal(threshold(np.array([1, 0.5, 0, 0.5])), np.array([1, 0, -1, 0]))
assert_array_equal(
    threshold(np.array([[0.5, 0.2, -0.3, 0.1], [1.7, -3.8, 0.5, 0.6]])), 
    np.array([[1, 1, -1, 1], [1, -1, 1, 1]]))
print("Success!")
"""Make sure a copy of the array is being returned, and that the original array is unmodified."""
x = np.array([[0.5, 0.2, -0.3, 0.1], [1.7, -3.8, 0.5, 0.6]])
y = threshold(x)
assert_array_equal(x, np.array([[0.5, 0.2, -0.3, 0.1], [1.7, -3.8, 0.5, 0.6]]))
assert_array_equal(y, np.array([[1, 1, -1, 1], [1, -1, 1, 1]]))
print("Success!")<jupyter_output><empty_output><jupyter_text>Array indexing The other type of fancy indexing is *array indexing*. Let's consider our average response across participants:<jupyter_code>data = np.load("data/experiment_data.npy")
avg_responses = np.mean(data, axis=1)
avg_responses<jupyter_output><empty_output><jupyter_text>And let's say we also know which element corresponds to which participant, through the following `participants` array:<jupyter_code>participants = np.load("data/experiment_participants.npy")
participants<jupyter_output><empty_output><jupyter_text>In other words, the first element of `avg_responses` corresponds to the first element of `participants` (so participant 45), the second element of `avg_responses` was given by participant 39, and so on.Let's say we wanted to know what participants had the largest average response, and what participants had the smallest average response. To do this, we might try sorting the responses:<jupyter_code>np.sort(avg_responses)<jupyter_output><empty_output><jupyter_text>However, we then don't know which responses correspond to which trials. A different way to do this would be to use `np.argsort`, which returns an array of indices corresponding to the sorted order of the elements, rather than the elements in sorted order:<jupyter_code>np.argsort(avg_responses)<jupyter_output><empty_output><jupyter_text>What this says is that element 18 is the smallest response, element 42 is the next smallest response, and so on, all the way to element 24, which is the largest response:<jupyter_code>avg_responses[18]
avg_responses[42]
avg_responses[24]<jupyter_output><empty_output><jupyter_text>To use fancy indexing, we can actually use this array of integers as an index. If we use it on the original array, then we will obtain the sorted elements:<jupyter_code>avg_responses[np.argsort(avg_responses)]<jupyter_output><empty_output><jupyter_text>And if we use it on our array of participants, then we can determine what participants had the largest and smallest responses:<jupyter_code>participants[np.argsort(avg_responses)]<jupyter_output><empty_output><jupyter_text>So, in this case, participant 10 had the smallest average response, while participant 47 had the largest average response.  From boolean to integer indices Sometimes, we want to use a combination of boolean and array indexing. For example, if we wanted to pull out just the responses for participant 2, a natural approach would be to use boolean indexing:<jupyter_code>participant_2_responses = data[participants == 'p_002']<jupyter_output><empty_output><jupyter_text>Another way that we could do this would be to determine the *index* of participant 2, and then use that to index into `data`. To do this, we can use a function called `np.argwhere`, which returns the indices of elements that are true:<jupyter_code>np.argwhere(participants == 'p_002')<jupyter_output><empty_output><jupyter_text>So in this case, we see that participant 2 corresponds to index 26.  Exercise: Averaging responses (2 points) Write a function that takes as arguments a participant id, the data, and the list of participant names, and computes the average response for the given participant. Occasionally we will ask you to raise an error if your function gets inputs that it's not expecting. As a reminder, to raise an error, you should use the raise keyword. For example, to raise a ValueError, you would do raise ValueError(message), where message is a string explaining specifically what the error was.<jupyter_code>def participant_mean(participant, data, participants):
    """Computes the mean response for the given participant. A ValueError
    should be raised if more than one participant has the given name.
    
    Hint: your solution should use `np.argwhere`, and can be done in
    four lines (including the return statement).
    
    Parameters
    ----------
    participant: string
        The name/id of the participant
    data: numpy.ndarray with shape (n, m)
        Rows correspond to participants, columns to trials
    participants: numpy.ndarray with shape(n,)
        A string array containing participant names/ids, corresponding to
        the rows of the `data` array.
        
    Returns
    -------
    float: the mean response of the participant over all trials
    
    """
    # YOUR CODE HERE
    raise NotImplementedError()
# add your own test cases in this cell!

"""Check for correct answers with the example experiment data."""
from numpy.testing import assert_allclose
data = np.load("data/experiment_data.npy")
participants = np.load("data/experiment_participants.npy")
assert_allclose(participant_mean('p_002', data, participants), 1857.7013113499095)
assert_allclose(participant_mean('p_047', data, participants), 1906.0651466520821)
assert_allclose(participant_mean('p_013', data, participants), 1718.4379910225193)
print("Success!")
"""Check for correct answers for some different data."""
data = np.arange(32).reshape((4, 8))
participants = np.array(['a', 'b', 'c', 'd'])
assert_allclose(participant_mean('a', data, participants), 3.5)
assert_allclose(participant_mean('b', data, participants), 11.5)
assert_allclose(participant_mean('c', data, participants), 19.5)
assert_allclose(participant_mean('d', data, participants), 27.5)
print("Success!")
"""Check that a ValueError is raised when the participant name is not unique."""
from nose.tools import assert_raises
data = np.arange(32).reshape((4, 8))
participants = np.array(['a', 'b', 'c', 'a'])
assert_raises(ValueError, participant_mean, 'a', data, participants)
print("Success!")<jupyter_output><empty_output><jupyter_text>--- Before finishing this notebook, remember to do the following steps:1. **Restart the kernel** (Kernel$\rightarrow$Restart)2. **Run all cells** (Cell$\rightarrow$Run All)3. **Save** (File$\rightarrow$Save and Checkpoint) After you have completed these three steps, ensure that the following cell has printed "No errors". If it has not printed "No errors", then your code has a bug in it and has thrown an error! Make sure you fix this error before finishing your notebook.<jupyter_code>print("No errors!")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08772080999569151, 'fraction_numerical': 0.01999138302455838, 'mean_word_length': 5.041644976574701, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '638802', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3465, 'n_tokens_neox': 3250, 'n_words': 1474, 'path': 'PS0/5 - Advanced NumPy.ipynb', 'repo_name': 'smitchaute/COGSCI-C131-PSYCH-123-FA20', 'size': '11605'}