<jupyter_start><jupyter_text>First day: list comprehensions and generators > List comprehensions and generators are in my top 5 favorite Python features leading to clean, robust and Pythonic code.<jupyter_code>from collections import Counter
import calendar
import itertools
import random
import re
import string

import requests<jupyter_output><empty_output><jupyter_text>List comprehensions Let's dive straight into a practical example. We all know how to use the classic for loop in Python, say I want to loop through a bunch of names title-casing each one:<jupyter_code>names = 'pybites <NAME> <NAME> guido'.split()
names
for name in names:
    print(name.title())<jupyter_output>Pybites
Mike
Bob
Julian
Tim
Sara
Guido<jupyter_text>Then I want to only keep the names that start with A-M, the `strings` module makes it easier (we love Python's standard library!):<jupyter_code>first_half_alphabet = list(string.ascii_lowercase)[:13]
first_half_alphabet
new_names = []
for name in names:
    if name[0] in first_half_alphabet:
        new_names.append(name.title())
new_names<jupyter_output><empty_output><jupyter_text>Feels verbose, not? If you don't know about list comprehensions you might start using them everywhere after seeing the next refactoring:<jupyter_code>new_names2 = [name.title() for name in names if name[0] in first_half_alphabet]
new_names2
assert new_names == new_names2<jupyter_output><empty_output><jupyter_text>From 4 to 1 lines of code, and it reads pretty well too. That's why we love and stick with Python! Here is another example I used recently to do a most common word count on Harry Potter. I used some list comprehensions to clean up the words before counting them:<jupyter_code>resp = requests.get('http://projects.bobbelderbos.com/pcc/harry.txt')
words = resp.text.lower().split()
words[:5]
cnt = Counter(words)
cnt.most_common(5)<jupyter_output><empty_output><jupyter_text>Hmm should not count stopwords, also:<jupyter_code>'-' in words<jupyter_output><empty_output><jupyter_text>Let's first clean up any non-alphabetic characters:<jupyter_code>words = [re.sub(r'\W+', r'', word) for word in words]
'-' in words
'the' in words<jupyter_output><empty_output><jupyter_text>Ok let's filter those stopwords out plus the empty strings caussed by the previous list comprehension:<jupyter_code>resp = requests.get('http://projects.bobbelderbos.com/pcc/stopwords.txt')
stopwords = resp.text.lower().split()
stopwords[:5]  
words = [word for word in words if word.strip() and word not in stopwords]
words[:5]
'the' in words<jupyter_output><empty_output><jupyter_text>Now it looks way better:<jupyter_code>cnt = Counter(words)
cnt.most_common(5)<jupyter_output><empty_output><jupyter_text>What's interesting here is that the first bit of the list comprehension can be an expression like `re.sub`. The final bit can be a compound statement: here we checked for a non-empty word (' ' -> `strip()` -> '' = `False` in Python) `and` we checked `word not in stopwords`. Again, a lot is going on in one line of code, but the beauty of it is that it is totally fine, because it reads like plain English :)  Generators A generator is a function that returns an iterator. It generates values using the `yield` keyword, when called with next() (a for loop does this implicitly), and it raises a `StopIteration` exception when there are no more values to generate. Let's see what this means with a very simple example:<jupyter_code>def num_gen():
    for i in range(5):
        yield i
        
gen = num_gen()
next(gen)
# note it takes off where we left it last statement
for i in gen:
    print(i)
# no more values to generate
next(gen)
# for catches the exception for us
for i in gen:
    print(i)<jupyter_output><empty_output><jupyter_text>> The `StopIteration` error appears because there are no more yield statements in the function. Calling next on the generator after this does not cause it to loop over and start again. - [Generators are Awesome, Learning by Example](https://pybit.es/generators.html) Since learning about generators, a common pattern I use is to build up my sequences:<jupyter_code>options = 'red yellow blue white black green purple'.split()
options<jupyter_output><empty_output><jupyter_text>My older code:<jupyter_code>def create_select_options(options=options):
    select_list = []
    
    for option in options:
        select_list.append(f'<option value={option}>{option.title()}</option>')
    
    return select_list
select_list_new = [f'<option value={option}>{option.title()}</option>' for option in options]
assert create_select_options(options) == select_list_new
from pprint import pprint as pp
pp(create_select_options())<jupyter_output>['<option value=red>Red</option>',
 '<option value=yellow>Yellow</option>',
 '<option value=blue>Blue</option>',
 '<option value=white>White</option>',
 '<option value=black>Black</option>',
 '<option value=green>Green</option>',
 '<option value=purple>Purple</option>']<jupyter_text>Using a generator you can write this in 2 lines of code - my newer code:<jupyter_code>def create_select_options_gen(options=options):    
    for option in options:
        yield f'<option value={option}>{option.title()}</option>'
print(create_select_options_gen())<jupyter_output><generator object create_select_options_gen at 0x10afca570><jupyter_text>Note that generators are _lazy_ so you need to explicitly consume them by iterating over them, for example by looping over them. Another way is to pass them into the `list()` constructor:<jupyter_code>list(create_select_options_gen())<jupyter_output><empty_output><jupyter_text>Specially when working with large data sets you definitely want to use generators. Lists can only get as big as they fit memory size. Generators are lazily evaluated meaning that they only hold a certain amount of data in memory at once. Just for the sake of giving Python something to do, let's calculate leap years for a million years, and compare performance of list vs generator:<jupyter_code># list
def leap_years_lst(n=1000000):
    leap_years = []
    for year in range(1, n+1):
        if calendar.isleap(year):
            leap_years.append(year)
    return leap_years

# generator
def leap_years_gen(n=1000000):
    for year in range(1, n+1):
        if calendar.isleap(year):
            yield year<jupyter_output><empty_output><jupyter_text>PRO  tip: [since Python 3.3](https://docs.python.org/3/whatsnew/3.3.html) you can use the `yield from` syntax.<jupyter_code># this had me waiting for a few seconds
%timeit -n1 leap_years_lst()
# this was instant
%timeit -n1 leap_years_gen()<jupyter_output>The slowest run took 4.72 times longer than the fastest. This could mean that an intermediate result is being cached.
870 ns ± 682 ns per loop (mean ± std. dev. of 7 runs, 1 loop each)<jupyter_text>That is pretty impressive. This is an important concept to know about because Big Data is here to stay!  Second day: practice Look at your code and see if you can refactor it to use list comprehensions. Same for generators. Are you building up a list somewhere where you could potentially use a generator? And/or exercise here, take this list of names:<jupyter_code>NAMES = ['<NAME>', '<NAME>', '<NAME>',
         '<NAME>', '<NAME>', '<NAME>',
         '<NAME>', '<NAME>', '<NAME>',
         'al pacino', 'br<NAME>itt', '<NAME>', 'brad pitt']<jupyter_output><empty_output><jupyter_text>Can you write a simple list comprehension to convert these names to title case (brad pitt -> Brad Pitt). Or reverse the first and last name?  Then use this same list and make a little generator, for example to randomly return a pair of names, try to make this work:    pairs = gen_pairs()    for _ in range(10):        next(pairs)Should print (values might change as random):    Arnold teams up with Brad    Alec teams up with JulianHave fun!  Third day: solution / simulate unix pipelines I hope yesterday's exercise was reasonably doable for you. Here are the answers in case you got stuck:<jupyter_code># list comprehension to title case names
[name.title() for name in NAMES]
# list comprehension to reverse first and last names
# using a helper here to show you that list comprehensions can be passed in functions!

def reverse_first_last_names(name):
    first, last = name.split()
    # ' '.join([last, first]) -- wait we have f-strings now (>= 3.6)
    return f'{last} {first}'

[reverse_first_last_names(name) for name in NAMES]
def gen_pairs():
    # again a list comprehension is great here to get the first names
    # and title case them in just 1 line of code (this comment took 2)
    first_names = [name.split()[0].title() for name in NAMES]
    while True:
        
        # added this when I saw Julian teaming up with Julian (always test your code!)
        first, second = None, None
        while first == second: 
            first, second = random.sample(first_names, 2)
        
        yield f'{first} teams up with {second}'
pairs = gen_pairs()
for _ in range(10):
    print(next(pairs))<jupyter_output>Alec teams up with Julbob
Keanu teams up with Bob
Keanu teams up with Julbob
Julian teams up with Arnold
Bob teams up with Alec
Matt teams up with Alec
Julbob teams up with Brad
Julian teams up with Alec
Julian teams up with Julbob
Julbob teams up with Julian<jupyter_text>Another way to get a slice of a generator is using `itertools.islice`:<jupyter_code>first_ten = itertools.islice(pairs, 10)
first_ten
list(first_ten)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09025727541121889, 'fraction_numerical': 0.007486292703500633, 'mean_word_length': 4.89863184079602, 'pattern_counts': {'":': 0, '<': 106, '<?xml version=': 0, '>': 112, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '17519', 'license': 'MIT', 'n_tokens_mistral': 2917, 'n_tokens_neox': 2824, 'n_words': 1124, 'path': 'days/16-18-listcomprehensions-generators/list-comprehensions-generators.ipynb', 'repo_name': 'billillib/100daysofcode-with-python-course', 'size': '9572'}