<jupyter_start><jupyter_text>The Core Language of Python: Part 3  For Loops and List Comprehensions A `for` loop can **iterate** over a list or tuple, performing calculations for each item in the sequence. Like `if` statements, `for` loops require a **colon** to terminate the first line and consistent **indentation** (typically four spaces) below it for the block of code that will be executed for each item in the sequence. Each item in the sequence is assigned a temporary variable name that can be used within the block. In the example below, this temporary variable is called `i`:<jupyter_code>my_list = [0, 1, 2, 3, 4, 5]

# print the square of each item in my_list
# - colon and indentation are important!
for i in my_list:
    print(i**2)<jupyter_output>0
1
4
9
16
25<jupyter_text>Be careful what name you give the iterator variable, since its value will be overwritten again and again with the items in the sequence. If the variable had a value before the `for` loop, it will be lost, which may not be what you intended.<jupyter_code>i = 'abc'
print(f'i at the start = {i}')

for i in my_list:
    print(i**2)

print(f'i at the end = {i}')<jupyter_output>i at the start = abc
0
1
4
9
16
25
i at the end = 5<jupyter_text>If you wanted to store a result from each step of the `for` loop in another list, one way you could do it is1. Initialize another variable as an empty list (`another_list = []`), and then2. Append a result to the new list in each step (`another_list.append`).For example:<jupyter_code>another_list = []
for i in my_list:
    another_list.append(i**2)

another_list<jupyter_output><empty_output><jupyter_text>If the calculation of the result (in this example, squaring `i`) is fairly simple, you can perform the same work using a more concise notation called **list comprehension**. The simplest version of list comprehension takes the form `[f(i) for i in my_list]`, where `f(i)` is some function or transformation of the list item `i`. Notice list comprehensions are enclosed in square brackets (`[]`) because they create lists. Here is a list comprehension equivalent to the example above:<jupyter_code># basic list comprehension
# - this means "square the item for each item in my_list"
another_list = [i**2 for i in my_list]
another_list<jupyter_output><empty_output><jupyter_text>All of the work is completed in a single line of code. Elegant!List comprehensions can be more complex than this. Suppose we modified the `for` loop to append a result only if the list item is an even number (`i % 2 == 0` means that `i` divided by 2 must have a remainder of 0):<jupyter_code>another_list = []
for i in my_list:
    if i % 2 == 0:
        # append only if i is even
        another_list.append(i**2)

# the squares of 0, 2, 4
another_list<jupyter_output><empty_output><jupyter_text>To do this with list comprehension, just add `if i % 2 == 0` to the end:<jupyter_code># list comprehension with conditional
# - this means "square the item for each item in my_list if it is even (otherwise skip it)"
another_list = [i**2 for i in my_list if i % 2 == 0]
another_list<jupyter_output><empty_output><jupyter_text>Suppose we modify the `for` loop further to perform a different calculation if the list item is an odd number:<jupyter_code>another_list = []
for i in my_list:
    if i % 2 == 0:
        # square if i is even
        another_list.append(i**2)
    else:
        # add 100 if i is odd
        another_list.append(i+100)

another_list<jupyter_output><empty_output><jupyter_text>This can be done with list comprehension by moving the `if i % 2 == 0` to an earlier position, just after `i**2`, and adding `else i+100`:<jupyter_code># list comprehension with complex conditional
# - this means "square the item if it is even, otherwise add 100 to it, for each item in my_list"
another_list = [i**2 if i%2==0 else i+100 for i in my_list]
another_list<jupyter_output><empty_output><jupyter_text>The results stored in `another_list` could be something other than a calculation using `i`. For example, strings:<jupyter_code># this means "store the string 'less than 2' if the item is less than 2, otherwise store '2 or greater', for each item in my_list"
another_list = ['less than 2' if i < 2 else '2 or greater' for i in my_list]
another_list<jupyter_output><empty_output><jupyter_text>While Loops A `while` loop is another way to repeatedly perform calculations. Whereas `for` loops execute code for each item in a sequence, `while` loops execute code for as long as a condition is true. For example:<jupyter_code>x = 0

while x < 5:
    print(x)
    x = x + 1

print(f'final value of x = {x}')<jupyter_output>0
1
2
3
4
final value of x = 5<jupyter_text>Generally, this means that the code within the `while` loop should take steps toward making the condition no longer true, even if it is unknown ahead of time how many steps that may require. In the simple example above, `x` was incremented each step until `x` was no longer less than 5. A more practical example would be a piece of code that reads a text file of unknown length one line at a time using a `while` loop that continues until a request for the next line yields nothing.Be warned: **If the condition never ceases to be true, the `while` loop will never stop**, which is probably not what you want!Try executing the code cell below, which will start an infinite loop because `x` is never incremented. You will see the icon in the left margin of the code cell spin and spin endlessly as the computer keeps executing the code within the `while` loop again and again, never stopping because the condition `x < 5` never stops being true. For this to end, you must **manually interrupt the code execution**, which you can do two ways:1. Click the spinning stop icon in the left margin, or2. Use the "Runtime" menu at the top of the page and click "Interrupt execution".Colab executes cells one at a time, so until you interrupt the execution of this cell, you will not be able to run any other code!<jupyter_code># this will run forever until interrupted!
x = 0
while x < 5:
    pass # do nothing<jupyter_output><empty_output><jupyter_text>Dictionaries Dictionaries are another important data type in Python. Dictionaries store **key-value pairs**, where each piece of data (the **value**) is assigned a name (the **key**) for easy access.Dictionaries are created using curly braces (`{}`). (This is different from the use of curly braces in f-strings!) Inside of the curly braces, key-value pairs are separated from one another by commas, and colons separate each key from its value. For example:<jupyter_code># create a new dictionary using curly braces {}
my_dict = {'genus': 'Aplysia', 'species': 'californica', 'mass': 150}
my_dict<jupyter_output><empty_output><jupyter_text>The syntax for extracting a piece of data from a dictionary is similar to indexing into lists. It uses square brackets after the dictionary name (not curly braces like you might guess), but instead of a number indicating position, a key should be provided.<jupyter_code># select items by key
my_dict['species']<jupyter_output><empty_output><jupyter_text>Like changing the value of an item in a list via its index, the value of an item in a dictionary can be changed via its key:<jupyter_code># change values by key
my_dict['mass'] = 300
my_dict<jupyter_output><empty_output><jupyter_text>New key-value pairs can be added to a dictionary the same way. In fact, you can start with an empty dictionary and build it up one key-value pair at a time:<jupyter_code>my_dict2 = {}

my_dict2['genus'] = 'Aplysia'
my_dict2['species'] = 'californica'
my_dict2['mass'] = 300

my_dict2<jupyter_output><empty_output><jupyter_text>Values can have any data type. Most basic data types are valid for keys too, but an important exception is lists: **lists are not allowed to be dictionary keys**. Tuples, on the other hand, are allowed to be keys. This is because keys must be immutable (uneditable), which is a property that tuples have but lists do not.<jupyter_code># lists cannot be keys, so this is NOT allowed
# - the error "unhashable type" is a consequence of the fact that lists are not immutable (they can be changed)
my_dict2[['x', 'y', 'z']] = [1, 2, 3]
# tuples can be keys, so this IS allowed
my_dict2[('x', 'y', 'z')] = [1, 2, 3]

my_dict2<jupyter_output><empty_output><jupyter_text>You can delete a key-value pair from a dictionary using the `del` keyword:<jupyter_code>del my_dict2['species']

my_dict2<jupyter_output><empty_output><jupyter_text>As a matter of fact, **`del` is how you unset any variable**:<jupyter_code>del my_dict2

# now my_dict2 is not defined
my_dict2<jupyter_output><empty_output><jupyter_text>Just like lists and tuples, `for` loops can iterate over a dictionary. In its simplest form, this actually iterates over the dictionary's keys. In the example below, we choose to use the name `k`, rather than `i`, for the temporary variable to reflect this. To access the value associated with key `k`, we must use `my_dict[k]`.<jupyter_code># iterate over keys
for k in my_dict:
    print(f'key: {k} --> value: {my_dict[k]}')<jupyter_output>key: genus --> value: Aplysia
key: species --> value: californica
key: mass --> value: 300<jupyter_text>The dictionary method `items()` returns a (special type of) list of tuples, where each tuple is a key-value pair:<jupyter_code># using list() here to simplify how the list of tuples is displayed
list(my_dict.items())<jupyter_output><empty_output><jupyter_text>When using a `for` loop to iterate over any list of tuples (or a list of lists, or a tuple of lists, or a tuple of tuples...) such as this, you can assign a temporary variable name to each item in the inner tuple/list. This is an example of what is called **unpacking**. For example:<jupyter_code>list_of_tuples = [
    ('a', 1),
    ('b', 2),
    ('c', 3),
]

for (letter, number) in list_of_tuples:
    print(letter, number)<jupyter_output>a 1
b 2
c 3<jupyter_text>In the example above, the parentheses around the iterator variables `(letter, number)` are actually optional, so the loop could be written without them:<jupyter_code>for letter, number in list_of_tuples:
    print(letter, number)<jupyter_output>a 1
b 2
c 3<jupyter_text>Using the list of tuples produced by `my_dict.items()` and unpacking each key-value tuple into `k,v`, we can write the `for` loop this way:<jupyter_code># iterate over key-value pairs
for k,v in my_dict.items():
    print(f'key: {k} --> value: {v}')<jupyter_output>key: genus --> value: Aplysia
key: species --> value: californica
key: mass --> value: 300<jupyter_text>Notice this does exactly the same thing as```pythonfor k in my_dict:    print(f'key: {k} --> value: {my_dict[k]}')```seen earlier, except the version using `for k,v in my_dict.items()` conveniently assigns the name `v` to the value of each key-value pair, so that `my_dict[k]` does not need to be typed out.Like list comprehensions, there exists a concise way for constructing dictionaries from a sequence, called **dictionary comprehension**. The syntax is similar, but a key and a value must be computed for each iteration, separated by a colon.To set up an example, here is a long way of constructing a dictionary of squares using a `for` loop, where the keys are string versions of the numbers:<jupyter_code>my_list = [0, 1, 2, 3, 4, 5]

squares = {}
for i in my_list:
    # store keys as strings and values as integers
    squares[str(i)] = i**2

squares<jupyter_output><empty_output><jupyter_text>Here is a dictionary comprehension that does the same thing. Note it is enclosed with curly braces because it produces a dictionary, and a colon separates a key and its value.<jupyter_code># basic dictionary comprehension
# - this means "pair a string version of the item with its square for each item in my_list"
squares = {str(i): i**2 for i in my_list}

squares<jupyter_output><empty_output><jupyter_text>Like list comprehension, conditionals are allowed for the values:<jupyter_code># dictionary comprehension with complex conditional
# - this means "pair a string version of the item with its square if it is even, otherwise with the item plus 100, for each item in my_list"
squares = {str(i): i**2 if i%2==0 else i+100 for i in my_list}

squares<jupyter_output><empty_output><jupyter_text>A Practical Example for Dictionaries and Comprehensions So how are dictionaries useful? There are countless ways, but let's look at one example. Previously we saw in the section titled "Lists vs. Tuples" that a list of tuples with consistent structure is useful because pieces of data have predictable indices. For example, with this definition of `species_data`, the genus of every entry always has index 0:<jupyter_code>species_data = [
    # genus, species, date named, common name
    ('Aplysia',     'californica', 1863, 'California sea hare'),
    ('Macrocystis', 'pyrifera',    1820, 'Giant kelp'),
    ('Pagurus',     'samuelis',    1857, 'Blueband hermit crab'),
]

# print every genus
for sp in species_data:
    print(sp[0])<jupyter_output>Aplysia
Macrocystis
Pagurus<jupyter_text>However, this approach requires that we memorize the meaning of each index (0 = genus, 1 = species, etc.). If we use dictionaries instead of tuples, we can access data by name, rather than arbitrary indices. To do this, we could convert every tuple into a dictionary, so that the whole data set is a list of dictionaries with identical keys.To demonstrate this, we could write everything out, like this:<jupyter_code>species_dicts = [
    {'genus': 'Aplysia',     'species': 'californica', 'year': 1863, 'common': 'California sea hare'},
    {'genus': 'Macrocystis', 'species': 'pyrifera',    'year': 1820, 'common': 'Giant kelp'},
    {'genus': 'Pagurus',     'species': 'samuelis',    'year': 1857, 'common': 'Blueband hermit crab'},
]<jupyter_output><empty_output><jupyter_text>However, if the `species_data` list already existed and was much longer than it is here, this would be a lot of extra work!Instead, we could use what we have learned to programmatically construct the list of dictionaries from the existing list of tuples using a `for` loop. Here's one way to do that which uses tuple unpacking for naming each of the four pieces of data in every tuple:<jupyter_code># create a new empty list
species_dicts = []

# for each tuple, unpack the 4 pieces of data into 4 temporary variables
for genus, species, year, common in species_data:

    # build a new dictionary for this species
    d = {'genus': genus, 'species': species, 'year': year, 'common': common}

    # append the dictionary to the new list
    species_dicts.append(d)

# display the new list of dictionaries
species_dicts<jupyter_output><empty_output><jupyter_text>Now the genus of every entry can be accessed using the key `'genus'` rather than index 0:<jupyter_code># print every genus
for sp in species_dicts:
    print(sp['genus'])<jupyter_output>Aplysia
Macrocystis
Pagurus<jupyter_text>If we want to be *really* clever, we can do the entire conversion in a single step by *constructing a dictionary inside a list comprehension*. For this, we need to first introduce another built-in function.The `zip()` function takes two or more sequences (e.g., lists or tuples) as inputs and groups the elements from each sequence in order. For example:<jupyter_code>list1 = ['a', 'b', 'c']
list2 = [1, 2, 3]

# using list() here to simplify how the result is displayed
list(zip(list1, list2))<jupyter_output><empty_output><jupyter_text>How can we use `zip()` to help us convert our list of tuples into a list of dictionaries? First, define a variable containing the dictionary keys:<jupyter_code>keys = ('genus', 'species', 'year', 'common')<jupyter_output><empty_output><jupyter_text>With this, it is possible to pair the values from one of the tuples with these keys. For example, if we just look at the first tuple:<jupyter_code>values = species_data[0]
list(zip(keys, values))<jupyter_output><empty_output><jupyter_text>Here we have a list of tuples, where each tuple is a key-value pair. This is just like what the `items()` function returns for a dictionary. From this, we could construct a dictionary from this first tuple using dictionary comprehension:<jupyter_code>{k:v for k,v in zip(keys, values)}<jupyter_output><empty_output><jupyter_text>Equivalently, because there is no extra manipulation of `k` or `v` here, we could use the built-in function `dict()` to convert the list of key-value pairs directly into a dictionary:<jupyter_code>dict(zip(keys, values))<jupyter_output><empty_output><jupyter_text>All we have to do now is generalize this to all tuples in `species_data`. We can use a list comprehension for this, which gives us the final expression that does the entire conversion in one step, from a list of tuples to a list of dictionaries:<jupyter_code>keys = ('genus', 'species', 'year', 'common')
species_dicts = [dict(zip(keys, values)) for values in species_data]
species_dicts<jupyter_output><empty_output><jupyter_text>This is much more elegant than writing out the list of dictionaries by hand!As a final proof of success, we can once again access the genus of every entry using a key:<jupyter_code># print every genus
for sp in species_dicts:
    print(sp['genus'])<jupyter_output>Aplysia
Macrocystis
Pagurus
{'fraction_non_alphanumeric': 0.08532285103185155, 'fraction_numerical': 0.010925487022371236, 'mean_word_length': 4.9986130374479885, 'pattern_counts': {'":': 0, '<': 143, '<?xml version=': 0, '>': 148, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '3819', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 5407, 'n_tokens_neox': 5130, 'n_words': 2421, 'path': 'notebooks/02.3-The-Core-Language-of-Python.ipynb', 'repo_name': 'jpgill86/python-for-neuroscientists', 'size': '17299'}