<jupyter_start><jupyter_text>Collections**Prerequisites**- [Core data types](https://datascience.quantecon.org/basics.html)  **Outcomes**- Ordered Collections      - Know what a list is and a tuple is    - Know how to tell a list from a tuple    - Understand the `range`, `zip` and `enumerate` functions    - Be able to use common list methods like `append`, `sort`,    and `reverse`    - Associative Collections      - Understand what a `dict` is    - Know the distinction between a dicts keys and values    - Understand when `dict`s are useful    - Be familiar with common `dict` methods    - Sets  (optional)      - Know what a set is    - Understand how a set differs from a list and a tuple    - Know when to use a set vs a list or a tuple    **Outline**  Contents- [Collections](Collections)    - [Ordered Collections](Ordered-Collections)    - [Associative Collections](Associative-Collections)    - [Exercises](Exercises)    Ordered Collections  ListsA Python list is an ordered collection of items.We can create lists using the following syntax ```python[item1, item2, ...,  itemN]``` where the `...` represents any number of additional items.Each item can be of any type.Let’s create some lists.<jupyter_code># created, but not assigned to a variable
[2.0, 9.1, 12.5]
# stored as the variable `x`
x = [2.0, 9.1, 12.5]
print("x has type", type(x))
x<jupyter_output><empty_output><jupyter_text>What Can We Do with Lists?We can access items in a list called `mylist` using `mylist[N]`where `N` is an integer.Note: Anytime that we use the syntax `x[i]` we are doing what iscalled indexing – it means that we are selecting a particular elementof a *collection* `x`.<jupyter_code>x[1]<jupyter_output><empty_output><jupyter_text>Wait? Why did `x[1]` return `9.1` when the first element in x isactually `2.0`?This happened because Python starts counting at zero!Lets repeat that one more time for emphasis **Python starts counting at zero**!To access the first element of x we must use `x[0]`:<jupyter_code>x[0]<jupyter_output><empty_output><jupyter_text>We can also determine how many items are in a list using the `len` function.<jupyter_code>len(x)<jupyter_output><empty_output><jupyter_text>What happens if we try to index with a number higher than the number ofitems in a list?<jupyter_code># uncomment the line below and run
# x[4]<jupyter_output><empty_output><jupyter_text>We can check if a list contains an element using the `in` keyword.<jupyter_code>2.0 in x
1.5 in x<jupyter_output><empty_output><jupyter_text>For our list `x`, other common operations we might want to do are…<jupyter_code>x.reverse()
x
number_list = [10, 25, 42, 1.0]
print(number_list)
number_list.sort()
print(number_list)<jupyter_output><empty_output><jupyter_text>Note that in order to `sort`, we had to have all elements in our listbe numbers (`int` and `float`), more on this [below](inhomogenous-lists).We could actually do the same with a list of strings. In this case, `sort`will put the items in alphabetical order.<jupyter_code>str_list = ["NY", "AZ", "TX"]
print(str_list)
str_list.sort()
print(str_list)<jupyter_output><empty_output><jupyter_text>The `append` method adds an element to the end of existing list.<jupyter_code>num_list = [10, 25, 42, 8]
print(num_list)
num_list.append(10)
print(num_list)<jupyter_output><empty_output><jupyter_text>However, if you call `append` with a list, it adds a `list` to the end,rather than the numbers in that list.<jupyter_code>num_list = [10, 25, 42, 8]
print(num_list)
num_list.append([20, 4])
print(num_list)<jupyter_output><empty_output><jupyter_text>To combine the lists instead…<jupyter_code>num_list = [10, 25, 42, 8]
print(num_list)
num_list.extend([20, 4])
print(num_list)<jupyter_output><empty_output><jupyter_text>> See exercise 1 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)  Lists of Different TypesWhile most examples above have all used a `list` witha single type of variable, this is not required.Let’s carefully make a small change to the first example: replace `2.0` with `2`<jupyter_code>x = [2, 9.1, 12.5]<jupyter_output><empty_output><jupyter_text>This behavior is identical for many operations you mightapply to a list.<jupyter_code>import numpy as np
x = [2, 9.1, 12.5]
np.mean(x) == sum(x)/len(x)<jupyter_output><empty_output><jupyter_text>Here we have also introduced a new [module](https://datascience.quantecon.org/basics.htmlmodules),[Numpy](https://datascience.quantecon.org/../scientific/index.html), which provides many functionsfor working with numeric data.Taking this further, we can put completely different types of elementsinside of a list.<jupyter_code># stored as the variable `x`
x = [2, "hello", 3.0]
print("x has type", type(x))
x<jupyter_output><empty_output><jupyter_text>To see the types of individual elements in the list:<jupyter_code>print(f"type(x[0]) = {type(x[0])}, type(x[0]) = {type(x[1])}, type(x[2]) = {type(x[2])}")<jupyter_output><empty_output><jupyter_text>While no programming limitations prevent this, you should be careful if you write codewith different numeric and non-numeric types in the same list.For example, if the types within the list cannot be compared, then how could you sort the elements of the list? (i.e. How do you determine whether the string “hello” is less than the integer 2, “hello” < 2?)<jupyter_code>x = [2, "hello", 3.0]
# uncomment the line below and see what happens!
# x.sort()<jupyter_output><empty_output><jupyter_text>A few key exceptions to this general rule are:- Lists with both integers and floating points are less error-prone  (since mathematical code using the list would work with both types).  - When working with lists and data, you may want to represent missing  values with a different type than the existing values.    The `range` FunctionOne function you will see often in Python is the `range` function.It has three versions:1. `range(N)`: goes from 0 to N-1  1. `range(a, N)`: goes from a to N-1  1. `range(a, N, d)`: goes from a to N-1, counting by d  When we call the `range` function, we get back something that has type `range`:<jupyter_code>r = range(5)
print("type(r)", type(r))<jupyter_output><empty_output><jupyter_text>To turn the `range` into a list:<jupyter_code>list(r)<jupyter_output><empty_output><jupyter_text>> See exercise 2 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)  What are Tuples?Tuples are very similar to lists and hold ordered collections of items.However, tuples and lists have three main differences:1. Tuples are created using parenthesis — `(` and `)` — instead of  square brackets — `[` and `]`.  1. Tuples are *immutable*, which is a fancy computer science word  meaning that they can’t be changed or altered after they are created.  1. Tuples and multiple return values  from functions are tightly connected, as we will see in [functions](https://datascience.quantecon.org/functions.html).<jupyter_code>t = (1, "hello", 3.0)
print("t is a", type(t))
t<jupyter_output><empty_output><jupyter_text>We can *convert* a list to a tuple by calling the `tuple` function ona list.<jupyter_code>print("x is a", type(x))
print("tuple(x) is a", type(tuple(x)))
tuple(x)<jupyter_output><empty_output><jupyter_text>We can also convert a tuple to a list using the list function.<jupyter_code>list(t)<jupyter_output><empty_output><jupyter_text>As with a list, we access items in a tuple `t` using `t[N]` where`N` is an int.<jupyter_code>t[0]  # still start counting at 0
t[2]<jupyter_output><empty_output><jupyter_text>> See exercise 3 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)Tuples (and lists) can be unpacked directly into variables.<jupyter_code>x, y = (1, "test")
print(f"x = {x}, y = {y}")<jupyter_output><empty_output><jupyter_text>This will be a convenient way to work with functions returningmultiple values, as well as within [comprehensions and loops](https://datascience.quantecon.org/control_flow.html).  List vs Tuple: Which to Use?Should you use a list or tuple?This depends on what you are storing, whether you might need to reorder the elements,or whether you’d addnew elements without a complete reinterpretation of theunderlying data.For example, take data representing the GDP (in trillions) and population(in billions) for China in 2015.<jupyter_code>china_data_2015 = ("China", 2015, 11.06, 1.371)

print(china_data_2015)<jupyter_output><empty_output><jupyter_text>In this case, we have used a tuple since: (a) ordering wouldbe meaningless; and (b) adding more data would require areinterpretation of the whole data structure.On the other hand, consider a list of GDP in China between2013 and 2015.<jupyter_code>gdp_data = [9.607, 10.48, 11.06]
print(gdp_data)<jupyter_output><empty_output><jupyter_text>In this case, we have used a list, since adding on a newelement to the end of the list for GDP in 2016 would makecomplete sense.Along these lines, collecting data on China for differentyears may make sense as a list of tuples (e.g. year, GDP,and population – although we will see better ways to store this sort of datain the [Pandas](https://datascience.quantecon.org/../pandas/index.html) section).<jupyter_code>china_data = [(2015, 11.06, 1.371), (2014, 10.48, 1.364), (2013, 9.607, 1.357)]
print(china_data)<jupyter_output><empty_output><jupyter_text>In general, a rule of thumb is to use a list unless you *need* to use a tuple.Key criteria for tuple use are when you want to:- ensure the *order* of elements can’t change  - ensure the actual values of the elements can’t  change  - use the collection as a key in a dict (we will learn what this  means [soon](https://datascience.quantecon.org/.html))    `zip` and `enumerate`Two functions that can be extremely useful are `zip` and `enumerate`.Both of these functions are best understood by example, so let’s seethem in action and then talk about what they do.<jupyter_code>gdp_data = [9.607, 10.48, 11.06]
years = [2013, 2014, 2015]
z = zip(years, gdp_data)
print("type(z)", type(z))<jupyter_output><empty_output><jupyter_text>To see what is inside `z`, let’s convert it to a list.<jupyter_code>list(z)<jupyter_output><empty_output><jupyter_text>Notice that we now have a list where each item is a tuple.Within each tuple, we have one item from each of the collections wepassed to the zip function.In particular, the first item in `z` contains the first item from`[2013, 2014, 2015]` and the first item from `[9.607, 10.48, 11.06]`.The second item in `z` contains the second item from each collectionand so on.We can access an element in this and then unpack the resultingtuple directly into variables.<jupyter_code>l = list(zip(years, gdp_data))
x, y = l[0]
print(f"year = {x}, GDP = {y}")<jupyter_output><empty_output><jupyter_text>Now let’s experiment with `enumerate`.<jupyter_code>e = enumerate(["a", "b", "c"])
print("type(e)", type(e))
e<jupyter_output><empty_output><jupyter_text>Again, we call `list(e)` to see what is inside.<jupyter_code>list(e)<jupyter_output><empty_output><jupyter_text>We again have a list of tuples, but this time, the first element in eachtuple is the *index* of the second tuple element in the initialcollection.Notice that the third item is `(2, 'c')` because`["a", "b", "c"][2]` is `'c'`> See exercise 4 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)An important quirk of some iterable types that are not lists (such as the above `zip`) is thatyou cannot convert the same type to a list twice.This is because `zip`, `enumerate`, and `range` produce what is called a generator.A generator will only produce each of its elements a single time, so if you call `list` on the samegenerator a second time, it will not have any elements to iterate over anymore.For more information, refer to the [Python documentation](https://docs.python.org/3/howto/functional.htmlgenerators).<jupyter_code>gdp_data = [9.607, 10.48, 11.06]
years = [2013, 2014, 2015]
z = zip(years, gdp_data)
l = list(z)
print(l)
m = list(z)
print(m)<jupyter_output><empty_output><jupyter_text>Associative Collections  DictionariesA dictionary (or dict) associates `key`s with `value`s.It will feel similar to a dictionary for words, where the keys are words andthe values are the associated definitions.The most common way to create a `dict` is to use curly braces — `{`and `}` — like this: ```python{"key1": value1, "key2": value2, ..., "keyN": valueN}``` where the `...` indicates that we can have any number of additionalterms.The crucial part of the syntax is that each key-value pair is written`key: value` and that these pairs are separated by commas — `,`.Let’s see an example using our aggregate data on China in 2015.<jupyter_code>china_data = {"country": "China", "year": 2015, "GDP" : 11.06, "population": 1.371}
print(china_data)<jupyter_output><empty_output><jupyter_text>Unlike our above example using a `tuple`, a `dict` allows us toassociate a name with each field, rather than having to remember theorder within the tuple.Often, code that makes a dict is easier to read if we put each`key: value` pair on its own line. (Recall our earlier comment onusing whitespace effectively to improve readability!)The code below is equivalent to what we saw above.<jupyter_code>china_data = {
    "country": "China",
    "year": 2015,
    "GDP" : 11.06,
    "population": 1.371
}<jupyter_output><empty_output><jupyter_text>Most often, the keys (e.g. “country”, “year”, “GDP”, and “population”)will be strings, but we could also use numbers (`int`, or`float`) or even tuples (or, rarely, a combination of types).The values can be **any** type and different from each other.> See exercise 5 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)This next example is meant to emphasize how values can be*anything* – including another dictionary.<jupyter_code>companies = {"AAPL": {"bid": 175.96, "ask": 175.98},
             "GE": {"bid": 1047.03, "ask": 1048.40},
             "TVIX": {"bid": 8.38, "ask": 8.40}}
print(companies)<jupyter_output><empty_output><jupyter_text>Getting, Setting, and Updating dict ItemsWe can now ask Python to tell us the value for a particular key by usingthe syntax `d[k]`,  where `d` is our `dict` and `k` is the key for which we want tofind the value.For example,<jupyter_code>print(china_data["year"])
print(f"country = {china_data['country']}, population = {china_data['population']}")<jupyter_output><empty_output><jupyter_text>Note: when inside of a formatting string, you can use `'` instead of `"` as aboveto ensure the formatting still works with the embedded code.If we ask for the value of a key that is not in the dict, we will get an error.<jupyter_code># uncomment the line below to see the error
# china_data["inflation"]<jupyter_output><empty_output><jupyter_text>We can also add new items to a dict using the syntax `d[new_key] = new_value`.Let’s see some examples.<jupyter_code>print(china_data)
china_data["unemployment"] = "4.05%"
print(china_data)<jupyter_output><empty_output><jupyter_text>To update the value, we use assignment in the same way (which willcreate the key and value as required).<jupyter_code>print(china_data)
china_data["unemployment"] = "4.051%"
print(china_data)<jupyter_output><empty_output><jupyter_text>Or we could change the type.<jupyter_code>china_data["unemployment"] = 4.051
print(china_data)<jupyter_output><empty_output><jupyter_text>> See exercise 6 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)  Common `dict` FunctionalityWe can do some common things with dicts.We will demonstrate them with examples below.<jupyter_code># number of key-value pairs in a dict
len(china_data)
# get a list of all the keys
list(china_data.keys())
# get a list of all the values
list(china_data.values())
more_china_data = {"irrigated_land": 690_070, "top_religions": {"buddhist": 18.2, "christian" : 5.1, "muslim": 1.8}}

# Add all key-value pairs in mydict2 to mydict.
# if the key already appears in mydict, overwrite the
# value with the value in mydict2
china_data.update(more_china_data)
china_data
# Get the value associated with a key or return a default value
# use this to avoid the NameError we saw above if you have a reasonable
# default value
china_data.get("irrigated_land", "Data Not Available")
china_data.get("death_rate", "Data Not Available")<jupyter_output><empty_output><jupyter_text>> See exercise 7 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)> See exercise 8 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)  Sets (Optional)Python has an additional way to represent collections of items: sets.Sets come up infrequently, but you should be aware of them.If you are familiar with the mathematical concept of sets, then you willunderstand the majority of Python sets already.If you don’t know the math behind sets, don’t worry: we’ll cover thebasics of Python’s sets here.A set is an *unordered* collection of *unique* elements.The syntax for creating a set uses curly bracket `{` and `}`. ```python{item1, item2, ..., itemN}``` Here is an example.<jupyter_code>s = {1, "hello", 3.0}
print("s has type", type(s))
s<jupyter_output><empty_output><jupyter_text>> See exercise 9 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)As with lists and tuples, we can check if something is `in` the setand check the set’s length:<jupyter_code>print("len(s) =", len(s))
"hello" in s<jupyter_output><empty_output><jupyter_text>Unlike lists and tuples, we can’t extract elements of a set `s` using`s[N]` where `N` is a number. ```python Uncomment the line below to see what happens s[1]``` This is because sets are not ordered, so the notion of getting thesecond element (`s[1]`) is not well defined.We add elements to a set `s` using `s.add`.<jupyter_code>s.add(100)
s
s.add("hello") # nothing happens, why?
s<jupyter_output><empty_output><jupyter_text>We can also do set operations.Consider the set `s` from above and the set`s2 = {"hello", "world"}`.- `s.union(s2)`: returns a set with all elements in either `s` or  `s2`  - `s.intersection(s2)`: returns a set with all elements in both `s`  and `s2`  - `s.difference(s2)`: returns a set with all elements in `s` that  aren’t in `s2`  - `s.symmetric_difference(s2)`: returns a set with all elements in  only one of `s` and `s2`  > See exercise 10 in the [*exercise list*](https://datascience.quantecon.org/exerciselist-0)As with tuples and lists, a `set` function can convert othercollections to sets.<jupyter_code>x = [1, 2, 3, 1]
set(x)
t = (1, 2, 3, 1)
set(t)<jupyter_output><empty_output><jupyter_text>Likewise, we can convert sets to lists and tuples.<jupyter_code>list(s)
tuple(s)<jupyter_output><empty_output><jupyter_text>Exercises**Exercise 1**In the first cell, try `y.append(z)`.In the second cell try `y.extend(z)`.Explain the behavior.HINT: When you are trying to explain use `y.append?` and `y.extend?` tosee a description of what these methods are supposed to do.<jupyter_code>y = ["a", "b", "c"]
z = [1, 2, 3]
# your code here
print(y)
y = ["a", "b", "c"]
z = [1, 2, 3]
# your code here
print(y)<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-0))**Exercise 2**Experiment with the other two versions of the `range` function.<jupyter_code># try list(range(a, N)) -- you pick `a` and `N`
# try list(range(a, N, d)) -- you pick `a`, `N`, and `d`<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-1))**Exercise 3**Verify that tuples are indeed immutable by attempting the following:- Changing the first element of `t` to be `100`  - Appending a new element `"!!"` to the end of `t` (remember with a  list `x` we would use `x.append("!!")` to do this  - Sorting `t`  - Reversing `t`<jupyter_code># change first element of t
# appending to t
# sorting t
# reversing t<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-2))**Exercise 4****Challenging** For the tuple `foo` below, use a combination of `zip`,`range`, and `len` to mimic `enumerate(foo)`.Verify that your proposed solution is correct by converting each to a listand checking equality with `==`.HINT: You can see what the answer should look like by starting with`list(enumerate(foo))`.<jupyter_code>foo = ("good", "luck!")<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-3))**Exercise 5**Create a new dict which associates stock tickers with its stock price.Here are some tickers and a price.- AAPL: 175.96  - GOOGL: 1047.43  - TVIX: 8.38<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-4))**Exercise 6**Look at the [World Factbook for Australia](https://www.cia.gov/-library/publications/the-world-factbook/geos/as.html)and create a dictionary with data containing the following types:float, string, integer, list, and dict.  Choose any data you wish.To confirm, you should have a dictionary that you identified via a key.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-5))**Exercise 7**Use Jupyter's help facilities to learn how to use the `pop` method toremove the key `"irrigated_land"` (and its value) from the dict.<jupyter_code># uncomment and use the Inspector or ?
#china_data.pop(<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-6))**Exercise 8**Explain what happens to the value you popped.Experiment with calling `pop` twice.<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-7))**Exercise 9**Try creating a set with repeated elements (e.g. `{1, 2, 1, 2, 1, 2}`).What happens?Why?<jupyter_code># your code here<jupyter_output><empty_output><jupyter_text>([*back to text*](https://datascience.quantecon.org/exercise-8))**Exercise 10**Test out two of the operations described above using the original set wecreated, `s`, and the set created below `s2`.<jupyter_code>s2 = {"hello", "world"}
# Operation 1
# Operation 2<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11790314617835825, 'fraction_numerical': 0.02064539293568511, 'mean_word_length': 5.621693907875185, 'pattern_counts': {'":': 22, '<': 226, '<?xml version=': 0, '>': 235, 'https://': 28, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '56', 'dir': 'jupyter-structured-clean-dedup', 'id': '136876', 'license': 'MIT', 'n_tokens_mistral': 7659, 'n_tokens_neox': 7196, 'n_words': 2626, 'path': 'code_notebooks/1.1-quantecon/.ipynb_checkpoints/collections-checkpoint.ipynb', 'repo_name': 'jacqueshbartlett/data-science-methods', 'size': '22281'}