<jupyter_start><jupyter_text>List comprehensions and generatorsList comprehensions are a succinct and efficient way of performing things across iterables (objects that can be iterated over) that are capable of collapsing for loops into a single of code. They can additionally be used to filter both the iterator itself, as well as the output depending on a condition.  The general structure of a list comprehension can be thought of as follows:- iterable - iterable variable that represents a member of the iterable- output variable (this is what is collected for the output)  `[output expression for iterable variable in iterable]`  A simple example of where this might be used is illustrated below in the creation of a new list that is the squared version of a list of numbers.<jupyter_code>numbers = [1, 3, 4, 12, 10, 19, 7, 2]
numbers_squared = [num ** 2 for num in numbers]
print(numbers_squared)<jupyter_output>[1, 9, 16, 144, 100, 361, 49, 4]<jupyter_text>Since list comprehensions work with all iterables, you could use this with other objects types such as range objects:<jupyter_code>cubed_numbers = [i ** 3 for i in range(15)]
print(cubed_numbers)
print(type(numbers_squared))
print(type(cubed_numbers))<jupyter_output><class 'list'>
<class 'list'><jupyter_text>You can see from the code above that both of the resulting list comprehensions have returned a list themselves. This doesn't always have to be the case, however, and you can alter the syntax to return other outputs such as dictionaries and generators.<jupyter_code>cubed_dict = {i: i ** 3 for i in range(1,11)}
print(cubed_dict)
print(type(cubed_dict))<jupyter_output>{1: 1, 2: 8, 3: 27, 4: 64, 5: 125, 6: 216, 7: 343, 8: 512, 9: 729, 10: 1000}
<class 'dict'><jupyter_text>In this example it has allowed me to the store the int as the key to the corresponding cubed value.   Nested list comprehensionsIt is possible to nest list comprehensions to generate the same output that would otherwise be achieved by nesting for loops within one another.<jupyter_code>matrix = []
for i in range(5):
    row = []
    for j in range(5):
        row.append(j)
    matrix.append(row)
matrix<jupyter_output><empty_output><jupyter_text>This isn't the most succinct or clean way to generate a 5x5 matrix. I will make some improvements before creating the same output using just a single list comprehension.<jupyter_code>matrix = []
for i in range(5):
    matrix.append([i for i in range(5)])
matrix<jupyter_output><empty_output><jupyter_text>Here I have simply replaced one variable definition and loop with a list comprehension that can generate the row `[0,1,2,3,4]` for me within the append.<jupyter_code>[[i for i in range(5)] for i in range(5)]<jupyter_output><empty_output><jupyter_text>We have then gone another level out and generated the correct number of iterations using another list comprehension. This can essentially be read as for each number in 1-5, make me a list of the numbers 0-4.   Advanced comprehensions and conditionals We can also include conditionals into our list comprehensions to further control its actions.<jupyter_code>[num if num % 2 == 0 else '' for num in range(1,11)]
[num for num in range(1,11) if num % 2 == 0]<jupyter_output><empty_output><jupyter_text>You can immediately notice the difference between the first and second outputs. This is dependent on where the conditional is placed within the list comprehension. The first filters the output depending on a conditonal; this is changing the output expression. The latter only passes those values that meet a certain condition in the iterable as it is iterated through. They can, of course, be used together.<jupyter_code>[num if num % 3 == 0 else '' for num in range(1,20) if num % 2 == 0]<jupyter_output><empty_output><jupyter_text>The above code only returns even number through filtering the iterable, and then limits the printed output to those also divisible by 3.   Generators Generators are very similar to list comprehensions. From a syntactical point of view, you must change the `[]` brackets to `()` parentheses.<jupyter_code>print(type([num for num in range(5)]))
print(type((num for num in range(5))))<jupyter_output><class 'list'>
<class 'generator'><jupyter_text>A `generator` object is like a list comprehension apart from that it doesn't store the list in memory. It doesn't actually produce the list, but rather produces an iterable that we are able to iterate over and allows us to create elements of the list as required. This is done using iterable syntax such as `next(generator)`<jupyter_code>generator_numbers = (num for num in range(10))
next(generator_numbers)
next(generator_numbers)<jupyter_output><empty_output><jupyter_text>As you call next on an iterable, in this case a generator, the lowest case is popped off. This means that you can no longer access that value again without first redefining the iterable again.<jupyter_code>generator_numbers = (num for num in range(10))
next(generator_numbers)<jupyter_output><empty_output><jupyter_text>This is sometimes known as *lazy evaluation* where the evaluation of the expression isn't actually performed until that value itself is needed.  This is useful when the list that you are working with is too large to work with in memory, so you are forced to generate those values as you need them through the `next()` method.   Generator functionsThese are functions that when called produce generator objects.  You define them as you normally would with the `def` keyword. Instead of `return`, however, you use the `yield` keyword to add them to the sequence that is retained within the generator object.<jupyter_code>def num_seq(n):
    """ Generate the values 0 to n"""
    i = 0
    while i < n:
        yield i
        i += 1
        
result = num_seq(100)
type(result)
[number for number in result if number % 5 == 0]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07062627637848877, 'fraction_numerical': 0.02042205582028591, 'mean_word_length': 5.147489539748954, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '255549', 'license': 'MIT', 'n_tokens_mistral': 1742, 'n_tokens_neox': 1599, 'n_words': 809, 'path': 'programming/list_comprehensions.ipynb', 'repo_name': 'willcanniford/python-learning', 'size': '5876'}