<jupyter_start><jupyter_text>Python Data Science Toolbox (Part 2)  1. Using iterators in PythonLand**You'll learn all about iterators and iterables, which you have already worked with when writing for loops. You'll learn some handy functions that will allow you to effectively work with iterators. And you’ll finish the chapter with a use case that is pertinent to the world of data science and dealing with large amounts of data—in this case, data from Twitter that you will load in chunks using iterators.**  Iterators vs Iterables- An ***iterable*** is an object that can return an *iterator*.- An ***iterator*** is an object that keeps state and produces the next value when you call `next()` on it.  Iterating over iterables (1)In this exercise, you will reinforce your knowledge about these by iterating over and printing from iterables and iterators.You are provided with a list of strings `flash`. You will practice iterating over the list by using a `for` loop. You will also create an iterator for the list and access the values from the iterator.- Create a `for` loop to loop over `flash` and print the values in the list. Use `person` as the loop variable.<jupyter_code># Create a list of strings: flash
flash = ['<NAME>', 'barry allen', 'wally west', 'bart allen']

# Print each list item in flash using a for loop
for person in flash:
    print(person)<jupyter_output><NAME>
barry allen
wally west
bart allen<jupyter_text>- Create an iterator for the list `flash` and assign the result to `superhero`.- Print each of the items from `superhero` using `next()` 4 times.<jupyter_code># Create an iterator for flash: superhero
superhero = iter(flash)

# Print each item from the iterator
print(next(superhero))
print(next(superhero))
print(next(superhero))
print(next(superhero))<jupyter_output><NAME>
barry allen
wally west
bart allen<jupyter_text>Iterating over iterables (2)Not all iterables are *actual lists*. A couple of examples that we looked at are *strings* and the use of the `range()` function. In this exercise, we will focus on the `range()` function.You can use `range()` in a `for` loop as *if* it's a list to be iterated over:```pythonfor i in range(5):    print(i)```Recall that `range()` doesn't actually create the list; instead, it creates a range object with an iterator that produces the values until it reaches the limit (in the example, until the value 4). If `range()` created the actual list, calling it with a value of 10100 may not work, especially since a number as big as that may go over a regular computer's memory. The value 10100 is actually what's called a **Googol** which is a 1 followed by a hundred 0s. That's a huge number!Your task for this exercise is to show that calling `range()` with 10100 won't actually pre-create the list. - Create an **iterator** object `small_value` over `range(3)` using the function `iter()`.<jupyter_code># Create an iterator for range(3): small_value
small_value = iter(range(3))

# Print the values in small_value
print(next(small_value))
print(next(small_value))
print(next(small_value))<jupyter_output>0
1
2<jupyter_text>- Using a `for` loop, iterate over `range(3)`, printing the value for every iteration. Use `num` as the loop variable.<jupyter_code># Loop over range(3) and print the values
for i in range(3):
    print(i)<jupyter_output>0
1
2<jupyter_text>- Create an **iterator** object `googol` over `range(10 ** 100)`.<jupyter_code># Create an iterator for range(10 ** 100): googol
googol = iter(range(10 ** 100))

# Print the first 5 values from googol
print(next(googol))
print(next(googol))
print(next(googol))
print(next(googol))
print(next(googol))<jupyter_output>0
1
2
3
4<jupyter_text>Iterators as function argumentsYou've been using the `iter()` function to get an iterator object, as well as the `next()` function to retrieve the values one by one from the iterator object.There are also functions that take iterators and iterables as arguments. For example, the `list()` and `sum()` functions return a list and the sum of elements, respectively.In this exercise, you will use these functions by passing an iterable from `range()` and then printing the results of the function calls. - Create a `range` object that would produce the values from 10 to 20 using `range()`. Assign the result to `values`.<jupyter_code># Create a range object: values
values = range(10, 21)

# Print the range object
print(values)<jupyter_output>range(10, 21)<jupyter_text>- Use the `list()` function to create a list of values from the range object `values`. Assign the result to `values_list`.<jupyter_code># Create a list of integers: values_list
values_list = list(values)

# Print values_list
print(values_list)<jupyter_output>[10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]<jupyter_text>- Use the `sum()` function to get the sum of the values from 10 to 20 from the range object `values`. Assign the result to `values_sum`.<jupyter_code># Get the sum of values: values_sum
values_sum = sum(values)

# Print values_sum
print(values_sum)<jupyter_output>165<jupyter_text>Using enumerate`enumerate()` returns an `enumerate` object that produces a sequence of tuples, and each of the tuples is an *index-value* pair.In this exercise, you are given a list of strings `mutants` and you will practice using `enumerate()` on it by printing out a list of tuples and unpacking the tuples using a `for` loop. - Create a list of tuples from `mutants` and assign the result to `mutant_list`. Make sure you generate the tuples using `enumerate()` and turn the result from it into a list using `list()`.<jupyter_code># Create a list of strings: mutants
mutants = ['<NAME>', 
            '<NAME>', 
            '<NAME>', 
            '<NAME>', 
            '<NAME>']

# Create a list of tuples: mutant_list
mutant_list = list(enumerate(mutants))

# Print the list of tuples
print(mutant_list)<jupyter_output>[(0, '<NAME>'), (1, '<NAME>'), (2, '<NAME>'), (3, '<NAME>'), (4, 'kitty pryde')]<jupyter_text>- Complete the first `for` loop by unpacking the tuples generated by calling `enumerate()` on `mutants`. Use `index1` for the index and `value1` for the value when unpacking the tuple.<jupyter_code># Unpack and print the tuple pairs
for index1, value1 in enumerate(mutants):
    print(index1, value1)<jupyter_output>0 <NAME>
1 <NAME>
2 <NAME>
3 <NAME>
4 kitty pryde<jupyter_text>- Complete the second `for` loop similarly as with the first, but this time change the starting index to start from `1` by passing it in as an argument to the `start` parameter of `enumerate()`. Use `index2` for the index and `value2` for the value when unpacking the tuple.<jupyter_code># Change the start index
for index2, value2 in enumerate(mutants, start=1):
    print(index2, value2)<jupyter_output>1 <NAME>
2 <NAME>
3 <NAME>
4 max eisenhardt
5 kitty pryde<jupyter_text>Using zipAnother interesting function that you've learned is `zip()`, which takes any number of iterables and returns a `zip` object that is an iterator of tuples. If you wanted to print the values of a `zip` object, you can convert it into a list and then print it. Printing just a `zip` object will not return the values unless you unpack it first. In this exercise, you will explore this for yourself.First, you will use `list()` and `zip()` on these lists to generate a list of tuples. Then, you will create a `zip` object using `zip()`. Finally, you will unpack this `zip` object in a `for` loop to print the values in each tuple. Observe the different output generated by printing the list of tuples, then the `zip` object, and finally, the tuple values in the `for` loop.<jupyter_code>mutants = ['<NAME>', '<NAME>', '<NAME>', '<NAME>', 'kitty pryde']

aliases = ['prof x', 'iceman', 'nightcrawler', 'magneto', 'shadowcat']

powers = ['telepathy', 'thermokinesis', 'teleportation', 'magnetokinesis', 'intangibility']<jupyter_output><empty_output><jupyter_text>- Using `zip()` with `list()`, create a list of *tuples* from the three lists `mutants`, `aliases`, and `powers` (in that order) and assign the result to `mutant_data`.<jupyter_code># Create a list of tuples: mutant_data
mutant_data = list(zip(mutants, aliases, powers))

# Print the list of tuples
print(mutant_data)<jupyter_output>[('<NAME>', 'prof x', 'telepathy'), ('bobby drake', 'iceman', 'thermokinesis'), ('<NAME>', 'nightcrawler', 'teleportation'), ('<NAME>', 'magneto', 'magnetokinesis'), ('kitty pryde', 'shadowcat', 'intangibility')]<jupyter_text>- Using `zip()`, create a *zip object* called `mutant_zip` from the three lists `mutants`, `aliases`, and `powers`.<jupyter_code># Create a zip object using the three lists: mutant_zip
mutant_zip = zip(mutants, aliases, powers)

# Print the zip object
print(mutant_zip)<jupyter_output><zip object at 0x00000296053BEA80><jupyter_text>- Complete the `for` loop by unpacking the `zip` object you created and printing the tuple values. Use `value1`, `value2`, `value3` for the values from each of `mutants`, `aliases`, and `powers`, in that order.<jupyter_code># Unpack the zip object and print the tuple values
for value1,value2,value3 in mutant_zip:
    print(value1, value2, value3)<jupyter_output><NAME> prof x telepathy
bobby drake iceman thermokinesis
<NAME> nightcrawler teleportation
max eisenhardt magneto magnetokinesis
kitty pryde shadowcat intangibility<jupyter_text>Using * and zip to 'unzip'Let's play around with `zip()` a little more. There is no *unzip* function for doing the reverse of what `zip()` does. We can, however, reverse what has been zipped together by using `zip()` with a little help from `*`!`*` unpacks an iterable such as a list or a tuple into positional arguments in a function call.In this exercise, you will use `*` in a call to `zip()` to unpack the tuples produced by `zip()`.<jupyter_code>mutants = ('<NAME>', '<NAME>', '<NAME>', '<NAME>', '<NAME>')

powers = ('telepathy', 'thermokinesis', 'teleportation', 'magnetokinesis', 'intangibility')<jupyter_output><empty_output><jupyter_text>- Create a `zip` object by using `zip()` on `mutants` and `powers`, in that order. Assign the result to `z1`.- Print the tuples in `z1` by unpacking them into positional arguments using the `*` operator in a `print()` call.<jupyter_code># Create a zip object from mutants and powers: z1
z1 = zip(mutants, powers)

# Print the tuples in z1 by unpacking with *
print(z1)<jupyter_output><zip object at 0x00000296053D3B80><jupyter_text>- Because the previous `print()` call would have exhausted the elements in `z1`, recreate the `zip` object you defined earlier and assign the result again to `z1`.- 'Unzip' the tuples in `z1` by unpacking them into positional arguments using the `*` operator in a `zip()` call. Assign the results to `result1` and `result2`, in that order.- The last `print()` statements prints the output of comparing `result1` to `mutants` and `result2` to `powers`. See if the unpacked `result1` and `result2` are equivalent to `mutants` and `powers`, respectively.<jupyter_code># Re-create a zip object from mutants and powers: z1
z1 = zip(mutants, powers)

# 'Unzip' the tuples in z1 by unpacking with * and zip(): result1, result2
result1, result2 = zip(*z1)

# Check if unpacked tuples are equivalent to original tuples
print(result1 == mutants)
print(result2 == powers)<jupyter_output>True
True<jupyter_text>Processing large amounts of Twitter dataSometimes, the data we have to process reaches a size that is too much for a computer's memory to handle. This is a common problem faced by data scientists. A solution to this is to process an entire data source chunk by chunk, instead of a single go all at once.In this exercise, you will do just that. You will process a large csv file of Twitter data in the same way that you processed `'tweets.csv'` in Bringing it all together exercises of the prequel course, but this time, working on it in chunks of 10 entries at a time.*Be aware that this is real data from Twitter and as such there is always a risk that it may contain profanity or other offensive content (in this exercise, and any following exercises that also use real Twitter data).* - Initialize an empty dictionary `counts_dict` for storing the results of processing the Twitter data.- Iterate over the `'tweets.csv'` file by using a `for` loop. Use the loop variable `chunk` and iterate over the call to `pd.read_csv()` with a `chunksize` of 10.- In the inner loop, iterate over the column `'lang'` in `chunk` by using a `for` loop. Use the loop variable `entry`.<jupyter_code>import pandas as pd

# Initialize an empty dictionary: counts_dict
counts_dict = {}

# Iterate over the file chunk by chunk
for chunk in pd.read_csv('tweets.csv', chunksize=10):

    # Iterate over the column in DataFrame
    for entry in chunk['lang']:
        if entry in counts_dict.keys():
            counts_dict[entry] += 1
        else:
            counts_dict[entry] = 1

# Print the populated dictionary
print(counts_dict)<jupyter_output>{'en': 97, 'et': 1, 'und': 2}<jupyter_text>Extracting information for large amounts of Twitter dataIt's good to know how to process a file in smaller, more manageable chunks, but it can become very tedious having to write and rewrite the same code for the same task each time. In this exercise, you will be making your code more *reusable* by putting your work in the last exercise in a *function* definition. - Define the function `count_entries()`, which has 3 parameters. The first parameter is `csv_file` for the filename, the second is `c_size` for the chunk size, and the last is `colname` for the column name.- Iterate over the file in `csv_file` file by using a `for` loop. Use the loop variable `chunk` and iterate over the call to `pd.read_csv()`, passing `c_size` to `chunksize`.- In the inner loop, iterate over the column given by `colname` in `chunk` by using a `for` loop. Use the loop variable `entry`.- Call the `count_entries()` function by passing to it the filename `'tweets.csv'`, the size of chunks `10`, and the name of the column to count, `'lang'`. Assign the result of the call to the variable `result_counts`.<jupyter_code># Define count_entries()
def count_entries(csv_file, c_size, colname):
    """Return a dictionary with counts of occurrences as value for each key."""
    
    # Initialize an empty dictionary: counts_dict
    counts_dict = {}

    # Iterate over the file chunk by chunk
    for chunk in pd.read_csv(csv_file, chunksize=c_size):

        # Iterate over the column in DataFrame
        for entry in chunk[colname]:
            if entry in counts_dict.keys():
                counts_dict[entry] += 1
            else:
                counts_dict[entry] = 1

    # Return counts_dict
    return counts_dict

# Call count_entries(): result_counts
result_counts = count_entries('tweets.csv', 10, 'lang')

# Print result_counts
print(result_counts)<jupyter_output>{'en': 97, 'et': 1, 'und': 2}<jupyter_text>---  2. List comprehensions and generators**In this chapter, you'll build on your knowledge of iterators and be introduced to list comprehensions, which allow you to create complicated lists—and lists of lists—in one line of code! List comprehensions can dramatically simplify your code and make it more efficient, and will become a vital part of your Python data science toolbox. You'll then learn about generators, which are extremely helpful when working with large sequences of data that you may not want to store in memory, but instead generate on the fly.**  Write a basic list comprehensionYou will write a list comprehension and identify the output that will be produced.The list:```pythondoctor = ['house', 'cuddy', 'chase', 'thirteen', 'wilson']```How would a list comprehension that produces a list of the **first character** of each string in `doctor` look like?<jupyter_code>doctor = ['house', 'cuddy', 'chase', 'thirteen', 'wilson']
[doc[0] for doc in doctor]<jupyter_output><empty_output><jupyter_text>List comprehension over iterablesList comprehensions can be built over iterables. Given the following objects below, which of these can we build list comprehensions over?```pythondoctor = ['house', 'cuddy', 'chase', 'thirteen', 'wilson']range(50)underwood = 'After all, we are nothing more or less than what we choose to reveal.'jean = '24601'flash = ['jay garrick', 'barry allen', 'wally west', 'bart allen']valjean = 24601``` 1. ~You can build list comprehensions over all the objects except the string of number characters `jean`.~2. ~You can build list comprehensions over all the objects except the string lists `doctor` and `flash`.~3. ~You can build list comprehensions over all the objects except `range(50)`.~4. You can build list comprehensions over all the objects except the integer object `valjean`.**Answer: 4.**(1) strings, even strings of number characters, are iterable. (2) lists of string objects are iterable. (3) you can build list comprehensions over `range(50)`. Writing list comprehensionsYour job in this exercise is to write a list comprehension that produces a list of the squares of the numbers ranging from 0 to 9.- Using the range of numbers from `0` to `9` as your iterable and `i` as your iterator variable, write a list comprehension that produces a list of numbers consisting of the squared values of `i`.<jupyter_code># Create list comprehension: squares
squares = [i ** 2 for i in range(0, 10)]
print(squares)<jupyter_output>[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]<jupyter_text>Nested list comprehensionsIn this exercise, you will be writing a list comprehension *within* another list comprehension, or nested list comprehensions. It sounds a little tricky, but you can do it!Let's step aside for a while from strings. One of the ways in which lists can be used are in representing multi-dimension objects such as **matrices**. Matrices can be represented as a list of lists in Python. For example a 5 x 5 matrix with values `0` to `4` in each row can be written as:```pythonmatrix = [[0, 1, 2, 3, 4],          [0, 1, 2, 3, 4],          [0, 1, 2, 3, 4],          [0, 1, 2, 3, 4],          [0, 1, 2, 3, 4]]```Your task is to recreate this matrix by using nested listed comprehensions. Recall that you can create one of the rows of the matrix with a single list comprehension. To create the list of lists, you simply have to supply the list comprehension as the **output expression** of the overall list comprehension:`[` `[`*output expression*`]` `for` *iterator variable* `in` *iterable*`]`Note that here, the **output expression** is itself a list comprehension. - In the inner list comprehension - that is, the **output expression** of the nested list comprehension - create a list of values from `0` to `4` using `range()`. Use `col` as the iterator variable.- In the **iterable** part of your nested list comprehension, use `range()` to count 5 rows - that is, create a list of values from `0` to `4`. Use `row` as the iterator variable; note that you won't be needing this variable to create values in the list of lists.<jupyter_code># Create a 5 x 5 matrix using a list of lists: matrix
matrix = [[col for col in range(0, 5)] for row in range(0, 5)]

# Print the matrix
for row in matrix:
    print(row)<jupyter_output>[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4]
[0, 1, 2, 3, 4]<jupyter_text>Using conditionals in comprehensions (1)You've been using list comprehensions to build lists of values, sometimes using operations to create these values.An interesting mechanism in list comprehensions is that you can also create lists with values that meet only a certain condition. One way of doing this is by using conditionals on iterator variables. In this exercise, you will do exactly that!Recall from the video that you can apply a conditional statement to test the iterator variable by adding an `if` statement in the optional *predicate expression* part after the `for` statement in the comprehension:`[` *output expression* `for` *iterator variable* `in` *iterable* `if` *predicate expression* `]`You will use this recipe to write a list comprehension for this exercise. You are given a list of strings `fellowship` and, using a list comprehension, you will create a list that only includes the members of `fellowship` that have 7 characters or more.<jupyter_code>fellowship = ['frodo', 'samwise', 'merry', 'aragorn', 'legolas', 'boromir', 'gimli']<jupyter_output><empty_output><jupyter_text>- Use `member` as the iterator variable in the list comprehension. For the conditional, use `len()` to evaluate the iterator variable. Note that you only want strings with 7 characters or more.<jupyter_code># Create list comprehension: new_fellowship
new_fellowship = [member for member in fellowship if len(member) >= 7]

# Print the new list
print(new_fellowship)<jupyter_output>['samwise', 'aragorn', 'legolas', 'boromir']<jupyter_text>Using conditionals in comprehensions (2)In the previous exercise, you used an `if` conditional statement in the *predicate expression* part of a list comprehension to evaluate an iterator variable. In this exercise, you will use an `if-else` statement on the *output expression* of the list.You will work on the same list, `fellowship` and, using a list comprehension and an `if-else` conditional statement in the output expression, create a list that keeps members of `fellowship` with 7 or more characters and replaces others with an empty string. Use `member` as the iterator variable in the list comprehension. - In the output expression, keep the string as-is **if** the number of characters is >= 7, **else** replace it with an empty string - that is, `''` or `""`.<jupyter_code># Create list comprehension: new_fellowship
new_fellowship = [member if len(member) >=7 else '' for member in fellowship]

# Print the new list
print(new_fellowship)<jupyter_output>['', 'samwise', '', 'aragorn', 'legolas', 'boromir', '']<jupyter_text>Dict comprehensionsComprehensions aren't relegated merely to the world of lists. There are many other objects you can build using comprehensions, such as dictionaries, pervasive objects in Data Science. You will create a dictionary using the comprehension syntax for this exercise. In this case, the comprehension is called a **dict comprehension**.Recall that the main difference between a *list comprehension* and a *dict comprehension* is the use of curly braces `{}` instead of `[]`. Additionally, members of the dictionary are created using a colon `:`, as in ` : `.You are given a list of strings `fellowship` and, using a **dict comprehension**, create a dictionary with the members of the list as the keys and the length of each string as the corresponding values. - Create a dict comprehension where the key is a string in `fellowship` and the value is the length of the string. Remember to use the syntax ` : ` in the output expression part of the comprehension to create the members of the dictionary. Use `member` as the iterator variable.<jupyter_code># Create dict comprehension: new_fellowship
new_fellowship = {member : len(member) for member in fellowship}

# Print the new dictionary
print(new_fellowship)<jupyter_output>{'frodo': 5, 'samwise': 7, 'merry': 5, 'aragorn': 7, 'legolas': 7, 'boromir': 7, 'gimli': 5}<jupyter_text>List comprehensions vs generatorsList comprehensions and generator expressions look very similar in their syntax, except for the use of parentheses `()` in generator expressions and brackets `[]` in list comprehensions.In this exercise, you will recall the difference between list comprehensions and generators. To help with that task, the following code:<jupyter_code># List of strings
fellowship = ['frodo', 'samwise', 'merry', 'aragorn', 'legolas', 'boromir', 'gimli']

# List comprehension
fellow1 = [member for member in fellowship if len(member) >= 7]

# Generator expression
fellow2 = (member for member in fellowship if len(member) >= 7)<jupyter_output><empty_output><jupyter_text>Select from the options below the best description for the difference between list comprehensions and generators.1. ~List comprehensions and generators are not different at all; they are just different ways of writing the same thing.~2. A list comprehension produces a list as output, a generator produces a generator object.3. ~A list comprehension produces a list as output that can be iterated over, a generator produces a generator object that can't be iterated over.~**Answer: 2.**<jupyter_code>print('fellow1:', type(fellow1), '-' ,fellow1, '\nfellow2:', type(fellow2), '-' ,fellow2)<jupyter_output>fellow1: <class 'list'> - ['samwise', 'aragorn', 'legolas', 'boromir'] 
fellow2: <class 'generator'> - <generator object <genexpr> at 0x0000021B6FF75A50><jupyter_text>Write your own generator expressionsIn this exercise, you will practice building generator expressions on your own.Recall that generator expressions basically have the same syntax as list comprehensions, except that it uses parentheses `()` instead of brackets `[]`. Furthermore, if you have ever iterated over a dictionary with `.items()`, or used the `range()` function, for example, you have already encountered and used generators before, without knowing it. When you use these functions, Python creates generators for you behind the scenes.Now, you will start simple by creating a generator object that produces numeric values. - Create a generator object that will produce values from `0` to `30`. Assign the result to `result` and use `num` as the iterator variable in the generator expression.- Print the first `5` values by using `next()` appropriately in `print()`.<jupyter_code># Create generator object: result
result = (i for i in range(0, 31))

# Print the first 5 values
print(next(result))
print(next(result))
print(next(result))
print(next(result))
print(next(result))<jupyter_output>0
1
2
3
4<jupyter_text>- Print the rest of the values by using a `for` loop to iterate over the generator object.<jupyter_code># Print the rest of the values
for value in result:
    print(value)<jupyter_output>5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30<jupyter_text>Changing the output in generator expressionsIn this exercise, you will push this idea a little further by adding to the output expression of a generator expression. Because generator expressions and list comprehensions are so alike in syntax, this should be a familiar task for you.You are given a list of strings `lannister` and, using a generator expression, create a generator object that you will iterate over to print its values.<jupyter_code>lannister = ['cersei', 'jaime', 'tywin', 'tyrion', 'joffrey']<jupyter_output><empty_output><jupyter_text>- Write a generator expression that will generate the **lengths** of each string in `lannister`. Use `person` as the iterator variable. Assign the result to `lengths`.- Supply the correct iterable in the `for` loop for printing the values in the generator object.<jupyter_code># Create a generator object: lengths
lengths = (len(person) for person in lannister)

# Iterate over and print the values in lengths
for value in lengths:
    print(value)<jupyter_output>6
5
5
6
7<jupyter_text>Build a generatorIn previous exercises, you've dealt mainly with writing generator expressions, which uses comprehension syntax. Being able to use comprehension syntax for generator expressions made your work so much easier.Now, not only are there generator expressions, there are *generator functions* as well. **Generator functions** are functions that, like generator expressions, yield a series of values, instead of returning a single value. A generator function is defined as you do a regular function, but whenever it generates a value, it uses the keyword `yield` instead of `return`.In this exercise, you will create a generator function with a similar mechanism as the generator expression you defined in the previous exercise:```pythonlengths = (len(person) for person in lannister)``` - Write the function header for the function `get_lengths()` that has a single parameter, `input_list`.- In the `for` loop in the function definition, `yield` the *length* of the strings in `input_list`.- Write the iterable part of the `for` loop for printing the values generated by the `get_lengths()` generator function. Supply the call to `get_lengths()`, passing in the list `lannister`.<jupyter_code># Define generator function get_lengths
def get_lengths(input_list):
    """Generator function that yields the length of the strings in input_list."""

    # Yield the length of a string
    for person in input_list:
        yield len(person)

# Print the values generated by get_lengths()
for value in get_lengths(lannister):
    print(value)<jupyter_output>6
5
5
6
7<jupyter_text>List comprehensions for time-stamped dataYou will now make use of what you've learned from this chapter to solve a simple data extraction problem. You will also be introduced to a data structure, the pandas **Series**, in this exercise. We won't elaborate on it much here, but what you should know is that it is a data structure that you will be working with a lot of times when analyzing data from pandas DataFrames. You can think of DataFrame columns as single-dimension arrays called Series.In this exercise, you will be using a list comprehension to extract the time from time-stamped Twitter data.<jupyter_code>import pandas as pd
tweets_df = pd.read_csv('tweets.csv')<jupyter_output><empty_output><jupyter_text>- Extract the column `'created_at'` from `tweets_df` and assign the result to `tweet_time`. Fun fact: the extracted column in `tweet_time` here is a Series data structure.- Create a list comprehension that extracts the time from each row in `tweet_time`. Each row is a string that represents a timestamp, and you will access the *12th to 19th characters* in the string to extract the time. Use `entry` as the *iterator variable* and assign the result to `tweet_clock_time`.<jupyter_code># Extract the created_at column from tweets_df: tweet_time
tweet_time = tweets_df['created_at']

# Extract the clock time: tweet_clock_time
tweet_clock_time = [entry[11:19] for entry in tweet_time]

# Print the extracted times
print(tweet_clock_time)<jupyter_output>['23:40:17', '23:40:17', '23:40:17', '23:40:17', '23:40:17', '23:40:17', '23:40:18', '23:40:17', '23:40:18', '23:40:18', '23:40:18', '23:40:17', '23:40:18', '23:40:18', '23:40:17', '23:40:18', '23:40:18', '23:40:17', '23:40:18', '23:40:17', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:17', '23:40:18', '23:40:18', '23:40:17', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:18', '23:40:19', '23:40:18', '23:40:18', '23:40:18', '23:40:19', '23:40:19', '23:40:19', '23:40:18', '23:40:19', '23:40:19', '23:40:19', '23:40:18', '23:40:19', '23:40:19', '23:40:19', '23:40:18', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23[...]<jupyter_text>Conditional list comprehensions for time-stamped dataYou've successfully extracted the data of interest, the time, from a pandas DataFrame. Let's tweak your work further by adding a conditional that further specifies which entries to select.In this exercise, you will be using a list comprehension to extract the time from time-stamped Twitter data. You will add a conditional expression to the list comprehension so that you only select the times in which `entry[17:19]` is equal to `'19'`. - Extract the column `'created_at'` from `tweets_df` and assign the result to `tweet_time`.- Create a list comprehension that extracts the time from each row in `tweet_time`. Each row is a string that represents a timestamp, and you will access the *12th to 19th characters* in the string to extract the time. Use `entry` as the iterator variable and assign the result to `tweet_clock_time`. Additionally, add a conditional expression that checks whether `entry[17:19]` is equal to `'19'`.<jupyter_code># Extract the created_at column from tweets_df: tweet_time
tweet_time = tweets_df['created_at']

# Extract the clock time: tweet_clock_time
tweet_clock_time = [entry[11:19] for entry in tweet_time if entry[17:19] == '19']

# Print the extracted times
print(tweet_clock_time)<jupyter_output>['23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19', '23:40:19']<jupyter_text>---  3. Bringing it all together**This chapter will allow you to apply your newly acquired skills toward wrangling and extracting meaningful information from a real-world dataset—the World Bank's World Development Indicators. You'll have the chance to write your own functions and list comprehensions as you work with iterators and generators to solidify your Python data science chops.**  Dictionaries for data scienceFor this exercise, you'll use what you've learned about the `zip()` function and combine two lists into a dictionary.These lists are actually extracted from a [bigger dataset file of world development indicators from the World Bank](https://datacatalog.worldbank.org/dataset/world-development-indicators). The first list `feature_names` contains header names of the dataset and the second list `row_vals` contains actual values of a row from the dataset, corresponding to each of the header names.<jupyter_code>feature_names = ['CountryName', 'CountryCode', 'IndicatorName', 
                 'IndicatorCode', 'Year', 'Value']
row_vals = ['Arab World', 'ARB', 'Adolescent fertility rate (births per 1,000 women ages 15-19)', 
            'SP.ADO.TFRT', '1960', '133.56090740552298']<jupyter_output><empty_output><jupyter_text>- Create a zip object by calling `zip()` and passing to it `feature_names` and `row_vals`. Assign the result to `zipped_lists`.- Create a dictionary from the `zipped_lists` zip object by calling `dict()` with `zipped_lists`. Assign the resulting dictionary to `rs_dict`.<jupyter_code># Zip lists: zipped_lists
zipped_lists = zip(feature_names, row_vals)

# Create a dictionary: rs_dict
rs_dict = dict(zipped_lists)

# Print the dictionary
print(rs_dict)<jupyter_output>{'CountryName': 'Arab World', 'CountryCode': 'ARB', 'IndicatorName': 'Adolescent fertility rate (births per 1,000 women ages 15-19)', 'IndicatorCode': 'SP.ADO.TFRT', 'Year': '1960', 'Value': '133.56090740552298'}<jupyter_text>Writing a function to help youSuppose you needed to repeat the same process done in the previous exercise to many, many rows of data. Rewriting your code again and again could become very tedious, repetitive, and unmaintainable.In this exercise, you will create a function to house the code you wrote earlier to make things easier and much more concise. Why? This way, you only need to call the function and supply the appropriate lists to create your dictionaries. - Define the function `lists2dict()` with two parameters: first is `list1` and second is `list2`.- Return the resulting dictionary `rs_dict` in `lists2dict()`.- Call the `lists2dict()` function with the arguments `feature_names` and `row_vals`. Assign the result of the function call to `rs_fxn`.<jupyter_code># Define lists2dict()
def lists2dict(list1, list2):
    """Return a dictionary where list1 provides the keys and list2 provides the values."""

    # Zip lists: zipped_lists
    zipped_lists = zip(list1, list2)

    # Create a dictionary: rs_dict
    rs_dict = dict(zipped_lists)

    # Return the dictionary
    return rs_dict

# Call lists2dict: rs_fxn
rs_fxn = lists2dict(feature_names, row_vals)

# Print rs_fxn
print(rs_fxn)<jupyter_output>{'CountryName': 'Arab World', 'CountryCode': 'ARB', 'IndicatorName': 'Adolescent fertility rate (births per 1,000 women ages 15-19)', 'IndicatorCode': 'SP.ADO.TFRT', 'Year': '1960', 'Value': '133.56090740552298'}<jupyter_text>Using a list comprehensionThis time, you're going to use the `lists2dict()` function you defined in the last exercise to turn a bunch of lists into a list of dictionaries with the help of a list comprehension.Your goal is to use a list comprehension to generate a list of dicts, where the keys are the header names and the values are the row entries.There is `row_lists`, which is a list of lists, where each sublist is a list of actual values of a row from the dataset.<jupyter_code>row_lists = [['Arab World', 'ARB', 'Adolescent fertility rate (births per 1,000 women ages 15-19)', 'SP.ADO.TFRT', '1960', '133.56090740552298'], ['Arab World', 'ARB', 'Age dependency ratio (% of working-age population)', 'SP.POP.DPND', '1960', '87.7976011532547'],['Arab World', 'ARB', 'Age dependency ratio, old (% of working-age population)', 'SP.POP.DPND.OL', '1960', '6.634579191565161'],['Arab World', 'ARB', 'Age dependency ratio, young (% of working-age population)', 'SP.POP.DPND.YG', '1960', '81.02332950839141'],['Arab World', 'ARB', 'Arms exports (SIPRI trend indicator values)', 'MS.MIL.XPRT.KD', '1960', '3000000.0'],['Arab World', 'ARB', 'Arms imports (SIPRI trend indicator values)', 'MS.MIL.MPRT.KD', '1960', '538000000.0'],['Arab World', 'ARB', 'Birth rate, crude (per 1,000 people)', 'SP.DYN.CBRT.IN', '1960', '47.697888095096395'],['Arab World', 'ARB', 'CO2 emissions (kt)', 'EN.ATM.CO2E.KT', '1960', '59563.9892169935'],['Arab World', 'ARB', 'CO2 emissions (metric tons per capita)', 'EN.ATM.CO2E.PC', '1960', '0.6439635478877049'],['Arab World', 'ARB', 'CO2 emissions from gaseous fuel consumption (% of total)', 'EN.ATM.CO2E.GF.ZS', '1960', '5.041291753975099'],['Arab World', 'ARB', 'CO2 emissions from liquid fuel consumption (% of total)', 'EN.ATM.CO2E.LF.ZS', '1960', '84.8514729446567'],['Arab World', 'ARB', 'CO2 emissions from liquid fuel consumption (kt)', 'EN.ATM.CO2E.LF.KT', '1960', '49541.707291032304'],['Arab World', 'ARB', 'CO2 emissions from solid fuel consumption (% of total)', 'EN.ATM.CO2E.SF.ZS', '1960', '4.72698138789597'],['Arab World', 'ARB', 'Death rate, crude (per 1,000 people)', 'SP.DYN.CDRT.IN', '1960', '19.7544519237187'],['Arab World', 'ARB', 'Fertility rate, total (births per woman)', 'SP.DYN.TFRT.IN', '1960', '6.92402738655897'],['Arab World', 'ARB', 'Fixed telephone subscriptions', 'IT.MLT.MAIN', '1960', '406833.0'],['Arab World', 'ARB', 'Fixed telephone subscriptions (per 100 people)', 'IT.MLT.MAIN.P2', '1960', '0.6167005703199'],['Arab World', 'ARB', 'Hospital beds (per 1,000 people)', 'SH.MED.BEDS.ZS', '1960', '1.9296220724398703'],['Arab World', 'ARB', 'International migrant stock (% of population)', 'SM.POP.TOTL.ZS', '1960', '2.9906371279862403'],['Arab World', 'ARB', 'International migrant stock, total', 'SM.POP.TOTL', '1960', '3324685.0']]<jupyter_output><empty_output><jupyter_text>Your goal is to use a list comprehension to generate a list of dicts, where the *keys* are the header names and the *values* are the row entries. - Inspect the contents of `row_lists` by printing the first two lists in `row_lists`.<jupyter_code># Print the first two lists in row_lists
print(row_lists[0])
print(row_lists[1])<jupyter_output>['Arab World', 'ARB', 'Adolescent fertility rate (births per 1,000 women ages 15-19)', 'SP.ADO.TFRT', '1960', '133.56090740552298']
['Arab World', 'ARB', 'Age dependency ratio (% of working-age population)', 'SP.POP.DPND', '1960', '87.7976011532547']<jupyter_text>- Create a list comprehension that generates a dictionary using `lists2dict()` for each sublist in `row_lists`. The keys are from the `feature_names` list and the values are the row entries in `row_lists`. Use `sublist` as your iterator variable and assign the resulting list of dictionaries to `list_of_dicts`.- Look at the first two dictionaries in `list_of_dicts` by printing them out.<jupyter_code># Turn list of lists into list of dicts: list_of_dicts
list_of_dicts = [lists2dict(feature_names, sublist) for sublist in row_lists]

# Print the first two dictionaries in list_of_dicts
print(list_of_dicts[0])
print(list_of_dicts[1])<jupyter_output>{'CountryName': 'Arab World', 'CountryCode': 'ARB', 'IndicatorName': 'Adolescent fertility rate (births per 1,000 women ages 15-19)', 'IndicatorCode': 'SP.ADO.TFRT', 'Year': '1960', 'Value': '133.56090740552298'}
{'CountryName': 'Arab World', 'CountryCode': 'ARB', 'IndicatorName': 'Age dependency ratio (% of working-age population)', 'IndicatorCode': 'SP.POP.DPND', 'Year': '1960', 'Value': '87.7976011532547'}<jupyter_text>Turning this all into a DataFrameYou've zipped lists together, created a function to house your code, and even used the function in a list comprehension to generate a list of dictionaries. That was a lot of work and you did a great job!You will now use all of these to convert the list of dictionaries into a pandas DataFrame. You will see how convenient it is to generate a DataFrame from dictionaries with the `DataFrame()` function from the pandas package. - Create a DataFrame from the list of dictionaries in `list_of_dicts` by calling `pd.DataFrame()`. Assign the resulting DataFrame to `df`.- Inspect the contents of `df` printing the head of the DataFrame. Head of the DataFrame `df` can be accessed by calling `df.head()`.<jupyter_code>import pandas as pd

# Turn list of lists into list of dicts: list_of_dicts
list_of_dicts = [lists2dict(feature_names, sublist) for sublist in row_lists]

# Turn list of dicts into a DataFrame: df
df = pd.DataFrame(list_of_dicts)

# Print the head of the DataFrame
df.head()<jupyter_output><empty_output><jupyter_text>Processing data in chunks (1)Sometimes, data sources can be so large in size that storing the entire dataset in memory becomes too resource-intensive. In this exercise, you will process the first 1000 rows of a file line by line, to create a dictionary of the counts of how many times each country appears in a column in the dataset. - Use `open()` to bind the csv file `'world_dev_ind.csv'` as `file` in the context manager.- Complete the `for` loop so that it iterates **1000** times to perform the loop body and process only the first 1000 rows of data of the file.<jupyter_code># Open a connection to the file
with open('world_dev_ind.csv') as file:

    # Skip the column names
    file.readline()

    # Initialize an empty dictionary: counts_dict
    counts_dict = {}

    # Process only the first 1000 rows
    for j in range(0, 1000):

        # Split the current line into a list: line
        line = file.readline().split(',')

        # Get the value for the first column: first_col
        first_col = line[0]

        # If the column value is in the dict, increment its value
        if first_col in counts_dict.keys():
            counts_dict[first_col] += 1

        # Else, add to the dict and set value to 1
        else:
            counts_dict[first_col] = 1

# Print the resulting dictionary
print(counts_dict)<jupyter_output>{'Arab World': 80, 'Caribbean small states': 77, 'Central Europe and the Baltics': 71, 'East Asia & Pacific (all income levels)': 122, 'East Asia & Pacific (developing only)': 123, 'Euro area': 119, 'Europe & Central Asia (all income levels)': 109, 'Europe & Central Asia (developing only)': 89, 'European Union': 116, 'Fragile and conflict affected situations': 76, 'Heavily indebted poor countries (HIPC)': 18}<jupyter_text>Writing a generator to load data in chunks (2)In the previous exercise, you processed a file line by line for a given number of lines. What if, however, you want to do this for the entire file?In this case, it would be useful to use generators. **Generators** allow users to [lazily evaluate](https://www.blog.pythonlibrary.org/2014/01/27/python-201-an-intro-to-generators/) data. This concept of lazy evaluation is useful when you have to deal with very large datasets because it lets you generate values in an efficient manner by *yielding* only chunks of data at a time instead of the whole thing at once.In this exercise, you will define a generator function `read_large_file()` that produces a generator object which yields a single line from a file each time `next()` is called on it. Note that when you open a connection to a file, the resulting file object is already a generator! So out in the wild, you won't have to explicitly create generator objects in cases such as this. However, for pedagogical reasons, we are having you practice how to do this here with the `read_large_file()` function. - In the function `read_large_file()`, read a line from `file_object` by using the method `readline()`. Assign the result to `data`.- In the function `read_large_file()`, `yield` the line read from the file `data`.- In the context manager, create a generator object `gen_file` by calling your generator function `read_large_file()` and passing `file` to it.- Print the first three lines produced by the generator object `gen_file` using `next()`.<jupyter_code># Define read_large_file()
def read_large_file(file_object):
    """A generator function to read a large file lazily."""

    # Loop indefinitely until the end of the file
    while True:

        # Read a line from the file: data
        data = file_object.readline()

        # Break if this is the end of the file
        if not data:
            break

        # Yield the line of data
        yield data
        
# Open a connection to the file
with open('world_dev_ind.csv') as file:

    # Create a generator object for the file: gen_file
    gen_file = read_large_file(file)

    # Print the first three lines of the file
    print(next(gen_file))
    print(next(gen_file))
    print(next(gen_file))<jupyter_output>CountryName,CountryCode,IndicatorName,IndicatorCode,Year,Value

Arab World,ARB,"Adolescent fertility rate (births per 1,000 women ages 15-19)",SP.ADO.TFRT,1960,133.5609074

Arab World,ARB,Age dependency ratio (% of working-age population),SP.POP.DPND,1960,87.79760115<jupyter_text>*Note that since a file object is already a generator, you don't have to explicitly create a generator object with your* `read_large_file()` *function. However, it is still good to practice how to create generators*  Writing a generator to load data in chunks (3)Great! You've just created a generator function that you can use to help you process large files.Now let's use your generator function to process the World Bank dataset like you did previously. You will process the file line by line, to create a dictionary of the counts of how many times each country appears in a column in the dataset. For this exercise, however, you won't process just 1000 rows of data, you'll process the entire dataset! - Bind the file `'world_ind_pop_data.csv'` to `file` in the context manager with `open()`.- Complete the `for` loop so that it iterates over the generator from the call to `read_large_file()` to process all the rows of the file.<jupyter_code># Initialize an empty dictionary: counts_dict
counts_dict = {}

# Open a connection to the file
with open('world_dev_ind.csv') as file:

    # Iterate over the generator from read_large_file()
    for line in read_large_file(file):

        row = line.split(',')
        first_col = row[0]

        if first_col in counts_dict.keys():
            counts_dict[first_col] += 1
        else:
            counts_dict[first_col] = 1

# Print            
print(counts_dict)<jupyter_output>{'CountryName': 1, 'Arab World': 80, 'Caribbean small states': 77, 'Central Europe and the Baltics': 71, 'East Asia & Pacific (all income levels)': 122, 'East Asia & Pacific (developing only)': 123, 'Euro area': 119, 'Europe & Central Asia (all income levels)': 109, 'Europe & Central Asia (developing only)': 89, 'European Union': 116, 'Fragile and conflict affected situations': 76, 'Heavily indebted poor countries (HIPC)': 99, 'High income': 119}<jupyter_text>Writing an iterator to load data in chunks (1)Another way to read data too large to store in memory in chunks is to read the file in as DataFrames of a certain length, say, 100. For example, with the pandas package (imported as `pd`), you can do `pd.read_csv(filename, chunksize=100)`. This creates an iterable **reader object**, which means that you can use `next()` on it.In this exercise, you will read a file in small DataFrame chunks with `read_csv()`. You're going to use the World Bank Indicators data `'world_ind_pop_data.csv'` to look at the urban population indicator for numerous countries and years. - Use `pd.read_csv()` to read in `'world_ind_pop_data.csv'` in chunks of size 10. Assign the result to `df_reader`.- Print the first two chunks from `df_reader`.<jupyter_code># Initialize reader object: df_reader
df_reader = pd.read_csv('world_ind_pop_data.csv', chunksize=10)

# Print two chunks
print(next(df_reader))
print(next(df_reader))<jupyter_output>CountryName CountryCode  Year  \
0                                 Arab World         ARB  1960   
1                     Caribbean small states         CSS  1960   
2             Central Europe and the Baltics         CEB  1960   
3    East Asia & Pacific (all income levels)         EAS  1960   
4      East Asia & Pacific (developing only)         EAP  1960   
5                                  Euro area         EMU  1960   
6  Europe & Central Asia (all income levels)         ECS  1960   
7    Europe & Central Asia (developing only)         ECA  1960   
8                             European Union         EUU  1960   
9   Fragile and conflict affected situations         FCS  1960   

   Total Population  Urban population (% of total)  
0      9.249590e+07                      31.285384  
1      4.190810e+06                      31.597490  
2      9.140158e+07                      44.507921  
3      1.042475e+09                      22.471132  
4      8[...]<jupyter_text>Writing an iterator to load data in chunks (2)In the previous exercise, you used `read_csv()` to read in DataFrame chunks from a large dataset. In this exercise, you will read in a file using a bigger DataFrame chunk size and then process the data from the first chunk.To process the data, you will create another DataFrame composed of only the rows from a specific country. You will then zip together two of the columns from the new DataFrame, `'Total Population'` and `'Urban population (% of total)'`. Finally, you will create a list of tuples from the zip object, where each tuple is composed of a value from each of the two columns mentioned. - Use `pd.read_csv()` to read in the file in `'world_ind_pop_data.csv'` in chunks of size `1000`. Assign the result to `urb_pop_reader`.- Get the **first** DataFrame chunk from the iterable `urb_pop_reader` and assign this to `df_urb_pop`.<jupyter_code># Initialize reader object: urb_pop_reader
urb_pop_reader = pd.read_csv('world_ind_pop_data.csv', chunksize=1000)

# Get the first DataFrame chunk: df_urb_pop
df_urb_pop = next(urb_pop_reader)

# Check out the head of the DataFrame
print(df_urb_pop.head())<jupyter_output>CountryName CountryCode  Year  \
0                               Arab World         ARB  1960   
1                   Caribbean small states         CSS  1960   
2           Central Europe and the Baltics         CEB  1960   
3  East Asia & Pacific (all income levels)         EAS  1960   
4    East Asia & Pacific (developing only)         EAP  1960   

   Total Population  Urban population (% of total)  
0      9.249590e+07                      31.285384  
1      4.190810e+06                      31.597490  
2      9.140158e+07                      44.507921  
3      1.042475e+09                      22.471132  
4      8.964930e+08                      16.917679<jupyter_text>- Select only the rows of `df_urb_pop` that have a `'CountryCode'` of `'CEB'`. To do this, compare whether `df_urb_pop['CountryCode']` is **equal** to `'CEB'` within the square brackets in `df_urb_pop[____]`.- Using `zip()`, zip together the `'Total Population'` and `'Urban population (% of total)'` columns of `df_pop_ceb`. Assign the resulting zip object to `pops`.<jupyter_code># Check out specific country: df_pop_ceb
df_pop_ceb = df_urb_pop[df_urb_pop['CountryCode'] == 'CEB']

# Zip DataFrame columns of interest: pops
pops = zip(df_pop_ceb['Total Population'], df_pop_ceb['Urban population (% of total)'])

# Turn zip object into list: pops_list
pops_list = list(pops)

# Print pops_list
print(pops_list)<jupyter_output>[(91401583.0, 44.5079211390026), (92237118.0, 45.206665319194), (93014890.0, 45.866564696018), (93845749.0, 46.5340927663649), (94722599.0, 47.2087429803526)]<jupyter_text>Writing an iterator to load data in chunks (3)You're getting used to reading and processing data in chunks by now. Let's push your skills a little further by adding a column to a DataFrame.Starting from the code of the previous exercise, you will be using a *list comprehension* to create the values for a new column `'Total Urban Population'` from the list of tuples that you generated earlier. Recall from the previous exercise that the first and second elements of each tuple consist of, respectively, values from the columns `'Total Population'` and `'Urban population (% of total)'`. The values in this new column `'Total Urban Population'`, therefore, are the product of the first and second element in each tuple. Furthermore, because the 2nd element is a percentage, you need to divide the entire result by `100`, or alternatively, multiply it by `0.01`.You will also plot the data from this new column to create a visualization of the urban population data. - Write a list comprehension to generate a list of values from `pops_list` for the new column `'Total Urban Population'`. The *output expression* should be the product of the first and second element in each tuple in `pops_list`. Because the 2nd element is a percentage, you also need to either multiply the result by `0.01` or divide it by `100`. In addition, note that the column `'Total Urban Population'` should only be able to take on integer values. To ensure this, make sure you cast the *output expression* to an integer with `int()`.- Create a *scatter* plot where the x-axis are values from the `'Year'` column and the y-axis are values from the `'Total Urban Population'` column.<jupyter_code>import matplotlib.pyplot as plt

# Code from previous exercise
urb_pop_reader = pd.read_csv('world_ind_pop_data.csv', chunksize=1000)
df_urb_pop = next(urb_pop_reader)
df_pop_ceb = df_urb_pop[df_urb_pop['CountryCode'] == 'CEB']
pops = zip(df_pop_ceb['Total Population'], 
           df_pop_ceb['Urban population (% of total)'])
pops_list = list(pops)

# Use list comprehension to create new DataFrame column 'Total Urban Population'
df_pop_ceb['Total Urban Population'] = [int(pops[0] * pops[1] * 0.01) for pops in pops_list]

# Plot urban population data
df_pop_ceb.plot(kind='scatter', x='Year', y='Total Urban Population')
plt.show()<jupyter_output><ipython-input-15-7d11ffe39854>:12: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_pop_ceb['Total Urban Population'] = [int(pops[0] * pops[1] * 0.01) for pops in pops_list]<jupyter_text>Writing an iterator to load data in chunks (4)In the previous exercises, you've only processed the data from the first DataFrame chunk. This time, you will aggregate the results over all the DataFrame chunks in the dataset. This basically means you will be processing the **entire** dataset now. This is neat because you're going to be able to process the entire large dataset by just working on smaller pieces of it! - Initialize an empty DataFrame data using `pd.DataFrame()`.- In the `for` loop, iterate over `urb_pop_reader` to be able to process all the DataFrame chunks in the dataset.- Using the method `append()` of the DataFrame `data`, append `df_pop_ceb` to `data`.<jupyter_code># Initialize reader object: urb_pop_reader
urb_pop_reader = pd.read_csv('world_ind_pop_data.csv', chunksize=1000)

# Initialize empty DataFrame: data
data = pd.DataFrame()

# Iterate over each DataFrame chunk
for df_urb_pop in urb_pop_reader:

    # Check out specific country: df_pop_ceb
    df_pop_ceb = df_urb_pop[df_urb_pop['CountryCode'] == 'CEB']

    # Zip DataFrame columns of interest: pops
    pops = zip(df_pop_ceb['Total Population'],
                df_pop_ceb['Urban population (% of total)'])

    # Turn zip object into list: pops_list
    pops_list = list(pops)

    # Use list comprehension to create new DataFrame column 'Total Urban Population'
    df_pop_ceb['Total Urban Population'] = [int(tup[0] * tup[1] * 0.01) for tup in pops_list]
    
    # Append DataFrame chunk to data: data
    data = data.append(df_pop_ceb)

# Plot urban population data
data.plot(kind='scatter', x='Year', y='Total Urban Population')
plt.show()<jupyter_output><ipython-input-16-524c8211b5d9>:21: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_pop_ceb['Total Urban Population'] = [int(tup[0] * tup[1] * 0.01) for tup in pops_list]<jupyter_text>Writing an iterator to load data in chunks (5)This is the last leg. You've learned a lot about processing a large dataset in chunks. In this last exercise, you will put all the code for processing the data into a single function so that you can reuse the code without having to rewrite the same things all over again.You're going to define the function `plot_pop()` which takes two arguments: the filename of the file to be processed, and the country code of the rows you want to process in the dataset.Because all of the previous code you've written in the previous exercises will be housed in `plot_pop()`, calling the function already does the following:Loading of the file chunk by chunk,Creating the new column of urban population values, andPlotting the urban population data.That's a lot of work, but the function now makes it convenient to repeat the same process for whatever file and country code you want to process and visualize! - Define the function `plot_pop()` that has two arguments: first is `filename` for the file to process and second is `country_code` for the country to be processed in the dataset.- Call `plot_pop()` to process the data for country code `'CEB'` in the file `'ind_pop_data.csv'`.- Call `plot_pop()` to process the data for country code `'ARB'` in the file `'ind_pop_data.csv'`.<jupyter_code># Define plot_pop()
def plot_pop(filename, country_code):

    # Initialize reader object: urb_pop_reader
    urb_pop_reader = pd.read_csv(filename, chunksize=1000)

    # Initialize empty DataFrame: data
    data = pd.DataFrame()
    
    # Iterate over each DataFrame chunk
    for df_urb_pop in urb_pop_reader:
        # Check out specific country: df_pop_ceb
        df_pop_ceb = df_urb_pop[df_urb_pop['CountryCode'] == country_code]

        # Zip DataFrame columns of interest: pops
        pops = zip(df_pop_ceb['Total Population'],
                    df_pop_ceb['Urban population (% of total)'])

        # Turn zip object into list: pops_list
        pops_list = list(pops)

        # Use list comprehension to create new DataFrame column 'Total Urban Population'
        df_pop_ceb['Total Urban Population'] = [int(tup[0] * tup[1] * 0.01) for tup in pops_list]
    
        # Append DataFrame chunk to data: data
        data = data.append(df_pop_ceb)

    # Plot urban population data
    data.plot(kind='scatter', x='Year', y='Total Urban Population')
    plt.show()

# Set the filename: fn
fn = 'world_ind_pop_data.csv'
# Call plot_pop for country code 'CEB'
plot_pop(fn, 'CEB')
# Call plot_pop for country code 'ARB'
plot_pop(fn, 'ARB')<jupyter_output><ipython-input-17-f5c074dbc96e>:23: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df_pop_ceb['Total Urban Population'] = [int(tup[0] * tup[1] * 0.01) for tup in pops_list]
{'fraction_non_alphanumeric': 0.09965034965034965, 'fraction_numerical': 0.039599551392004224, 'mean_word_length': 4.7103974383123, 'pattern_counts': {'":': 0, '<': 212, '<?xml version=': 0, '>': 217, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '53', 'dir': 'jupyter-structured-clean-dedup', 'id': '197205', 'license': 'FSFAP', 'n_tokens_mistral': 19604, 'n_tokens_neox': 17670, 'n_words': 7933, 'path': 'Data Analyst with Python/07_Python_Data_Science_Toolbox/07_2_Python Data Science Toolbox(part2).ipynb', 'repo_name': 'brendensong/DataCamp', 'size': '60839'}