<jupyter_start><jupyter_text>![tstaunton.jpg](attachment:tstaunton.jpg)*** List comprehension*** In the previous lessons we've used a lot of **for loops**, have you noticed that they can be inefficient when working with lists? There is another way to work with lists that's more efficient and takes only one line of code. It's done by using **list comprehensions**. The syntax uses square brackets where you place the values you wish to create also called the output expression. This is followed by a **for clause** that refers to the original list. As always this is better explained with an example.<jupyter_code># Create a list of numbers
numbers = [1,2,3,4,5,6]

# Create a list comprehension, new_numbers
new_numbers = [num for num in numbers] 

print(new_numbers)<jupyter_output>[1, 2, 3, 4, 5, 6]<jupyter_text>See how much more efficient this is when compared to a for loop.<jupyter_code># Create a list of numbers
numbers = [1,2,3,4,5,6]

new_numbers = []

for num in numbers:
    new_numbers.append(num + 1)
    
print(new_numbers)<jupyter_output>[2, 3, 4, 5, 6, 7]<jupyter_text>List comprehensions can be used over any iterable. Here is an example using the **range()** object.<jupyter_code>result = [num for num in range(11)]

print(result)<jupyter_output>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]<jupyter_text>In a previous lesson we looked at nested for loops. List comprehension can be used to replace these as well.<jupyter_code>combs = []

for x in [1,2,3]:
    for y in [3,1,4]:
        if x != y:
            combs.append((x,y))
        
combs<jupyter_output><empty_output><jupyter_text>Do you think that the same can be done with list comprehension? Within the square brackets place the desired output expression followed by the two required for loop clauses.<jupyter_code>combs2 = [(x,y) for x in [1,2,3] for y in [3,1,4] if x != y]
combs2<jupyter_output><empty_output><jupyter_text>If you are struggling to read this list comprehension you're not alone. In the above code example readability has been sacrificed. List comprehensions are something that only get easier with practice. In cases like the above it will be your decision to use list comprehension.  Using conditionals in comprehensionsIn the example below we have squared the values in **range(10)** and outputted the results only under the condition that the result is even. We do this buy using the **modulo operator (%)** which outputs the reminder of a division. In the case below we are dividing the squared values in range(10) by 2 and if the reminder is 0, meaning that it is an even number, it is outputted to a list.<jupyter_code>[num ** 2 for num in range(10) if num % 2 == 0] # Even numbers<jupyter_output><empty_output><jupyter_text>We can also use conditional operators within the list comprehension on the output expression. In the example below, for only an even integer we output its square. In any other case signified by the else condition for odd integers we output 0.<jupyter_code>[num ** 2 if num % 2 == 0 else 0 for num in range(10)]<jupyter_output><empty_output><jupyter_text>Using dict comprehensions we can create new dictionaries. The syntax is similar to list comprehension. There are however two main differences, firstly, the dictionary is formed with curly braces and secondly the key and value are separated by a colon in the output expression.<jupyter_code>pos_neg = {num: -num for num in range(10)}
pos_neg<jupyter_output><empty_output><jupyter_text>In this example we have created a dictionary with keys of positive integers and the corresponding values with the respective negative integers. An interesting mechanism in list comprehensions is that you can also create lists with values that meet only a certain condition. One way of doing this is by using conditionals on iterator variables.<jupyter_code># Create a list of strings: salaries
salaries = [20000, 30000, 40000, 50000, 60000, 70000]

# Create list comprehension: large_salaries
large_salaries = [value for value in salaries if value >= 45000]

# Print the new list
print(large_salaries)<jupyter_output>[50000, 60000, 70000]<jupyter_text>GeneratorsGenerators are related to comprehensions which is why we are talking about them here and now. Take a look below at the first line of code which is a list comprehension.<jupyter_code># List comprehension
[2 * num for num in range(10)]
# Use () instead
(2 * num for num in range(10))<jupyter_output><empty_output><jupyter_text>Straight away you'll notice the difference in both outputs. A list comprehension produces a list as output, a generator produces a **generator object**.So what is a generator? It's similar to a list comprehension except for the fact that it does not store the list in memory. But it is an object that we can iterate over to produce elements of a list as required.<jupyter_code># Looping over a generator expression produces the elements of a list
result = (num for num in range(10))

for num in result:
    print(num)<jupyter_output>0
1
2
3
4
5
6
7
8
9<jupyter_text>Like any other iterator we can pass a generator to the function **next()** so that we can iterate through its elements.<jupyter_code>result = (num for num in range(6))

print(next(result))

print(next(result))
print(next(result))<jupyter_output>0
1
2<jupyter_text>This is what's known as **lazy evaluation**, meaning that the evaluation of the expression is delayed until its value is needed. This is good when working with large sequences and you do not want to store the entire list in memory which is what lists do. Instead you can access elements as and when needed.  It's worth pointing out, anything that can be done with list comprehension can also be done with generators.<jupyter_code># Apply conditional to a generator expression
even_nums1 = (num for num in range(10) if num % 2 == 0)
print(list(even_nums1))<jupyter_output>[0, 2, 4, 6, 8]<jupyter_text>Generator functionsWe've already seen that generators are their own type. So generator functions are functions that when called produce generator objects. They are created with the same syntax used when creating any other function but instead of return values using the return keyword they yield sequences of values using the **yield** keyword.Take a look.<jupyter_code>def num_sequence(n):
    """Generate values from 0 to n."""
    i = 0
    while i < n:
        yield i
        i += 1
        
# Call the function        
result = num_sequence(10)

# Out the values it yields with a for loop
for item in result:
    print(item)<jupyter_output>0
1
2
3
4
5
6
7
8
9
{'fraction_non_alphanumeric': 0.0709480122324159, 'fraction_numerical': 0.02614678899082569, 'mean_word_length': 4.788495575221239, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '279662', 'license': 'Apache-2.0', 'n_tokens_mistral': 1987, 'n_tokens_neox': 1824, 'n_words': 930, 'path': 'Learn-python/Part 3 - Functions, Iterators & Generators/05-List comprehension.ipynb', 'repo_name': 'ATTO-DT/DATA-SCIENCE-PROJECT', 'size': '6540'}