<jupyter_start><jupyter_text>Software Carpentry Workshop - SIEPR Fellows ForewordThis notebook contains code samples and examples shown during the Software Carpentry workshop for SIEPR fellows, taught on July 10th 2019. These samples and examples assume basic knowledge of Python. For questions, suggestions, and feedback, email me at joaor _squiggly thing_ stanford _point_ edu.The contents of the workbook are based on the materials of the _Programming and Plotting in Python_ lesson, available [here](https://swcarpentry.github.io/python-novice-gapminder/). OverviewThis lesson is divided in four episodes:* Libraries* Reading Tabular Data into _pandas_ DataFrames* Indexing, Selecting, and Grouping _pandas_ DataFrames* Creating plots from datasets using _matplotlib_  LibrariesPart of Python's popularity stems from the fact that by default, any installation of the language comes with a large collection of _libraries_ or _modules_ - the Python Standard Library. Both terms refer to code, which can include variables, functions, or classes, that programmers like you can (re-)use when writing code. <NAME>, the creator of the Perl programming language, defined _laziness_ as one of the three great virtues of programmers. Indeed, modules are a great example of such laziness -previously written code that you can just use.The differences between a library and a module are subtle and mostly linguistic: a library or package (roughly synonymous) may contain multiple modules. For example, the library _matplotlib_, which we will learn at the end of this lesson, is often used through its module _pyplot_. However, you will see that these terms are used interchangeably, even in the Python documentation (The library reference lists all the modules of the Python language). I will use both terms interchangeably throughout the notebook.More importantly, libraries save you both time and effort. As a scientist, your time is best spent on analysing data instead of on re-writing code to load data from files in CSV format. Libraries also tend to be more robust and more optimized than code you would write yourself, specially if they are used by a lot of people. After all, nobody would use a slow and buggy library - a sort of natural selection law for computer code. Nevertheless, it __always__ pays off to validate any code you did not write on a known sample of data.So, on to coding. Code To import a library, type use the `import` statement followed by the name of the library, e.g.:<jupyter_code>import this  # prints the so-called 'Zen of Python'
# Let's import a slightly more useful library
import math<jupyter_output><empty_output><jupyter_text>After importing a module, you can use `print` to check that it really worked. The result is the 'string representation' of the module. For `math`, it reads 'built-in', since it's indeed built in Python itself! For free!<jupyter_code>print(math)<jupyter_output><module 'math' (built-in)><jupyter_text>If you try to import a module that your Python interpreter cannot find or load, you will get an error message of the type `ModuleNotFoundError` or `ImportError`. If this happens, check that:1. You typed the module name correctly - typos are probably 75% of import errors.2. You evaluated the cell containing the `import` statement when using Jupyter notebooks.2. You actually installed what you are trying to import, specially if it a third-party module (e.g. numpy, pandas)3. You are in the right virtual environment, if using venv or conda.<jupyter_code>import whatisthis<jupyter_output><empty_output><jupyter_text>To list the contents of a library, we can use eitherthe `help()` or the `dir()` functions, with the library itselfas the only parameter.`dir()` gives a simple list of all members of the library.It also included the so-called 'dunder' (double-underscore)methods. If a method's name starts with one or more underscores,it means you, as a user, should not call it directly.It's a Python convention to signal internal/accessory methodsthat the object (library, function, variable, etc) calls todo its job.The `help()` function prints a much nicer listing,because it includes all the documentation the librarydeveloper wrote to accompany the code!<jupyter_code>dir(math)
help(math)<jupyter_output>Help on built-in module math:

NAME
    math

DESCRIPTION
    This module is always available.  It provides access to the
    mathematical functions defined by the C standard.

FUNCTIONS
    acos(x, /)
        Return the arc cosine (measured in radians) of x.
    
    acosh(x, /)
        Return the inverse hyperbolic cosine of x.
    
    asin(x, /)
        Return the arc sine (measured in radians) of x.
    
    asinh(x, /)
        Return the inverse hyperbolic sine of x.
    
    atan(x, /)
        Return the arc tangent (measured in radians) of x.
    
    atan2(y, x, /)
        Return the arc tangent (measured in radians) of y/x.
        
        Unlike atan(y/x), the signs of both x and y are considered.
    
    atanh(x, /)
        Return the inverse hyperbolic tangent of x.
    
    ceil(x, /)
        Return the ceiling of x as an Integral.
        
        This is the smallest integer >= x.
    
    copysign(x, y, /)
        Return a float with the magnitude (absolute value) of[...]<jupyter_text>After you import a module, you have access to all its members but they are part of the so-called module's namespace. To access this namespace, use the `library.member` notation, the same way you called for instance the `split()` method for a string, or the `close` method for files.Let's start by accessing the value of `pi` stored in the `math` module.<jupyter_code>print(math.pi)
print(type(math.pi))  # math.pi is just a float someone once wrote in the module, nothing fancy!<jupyter_output>3.141592653589793
<class 'float'><jupyter_text>You can assign the value of `math.pi` to a variable of your choice, to make it easier or more practical to access it.<jupyter_code>pi = math.pi
print(pi)<jupyter_output><empty_output><jupyter_text>It is also possible to import specific members of a large module, which can be useful when part of a library requires some pre-processing to use or some dependencies that we did not install. To do so, use the `from ... import ...` syntax.<jupyter_code>from math import pi
print(pi)
print(math.pi == pi)  # this should be True
from math import pi, cos
print(cos(pi))  # cos() is a function of the math module to calcule cosines
from math import *  # this notation imports EVERYTHING from the module into the so-called global namespace.
print(pi)
print(sin(pi))<jupyter_output>3.141592653589793
1.2246467991473532e-16<jupyter_text>While this might seem extremely useful and a time saver - writing `pi` instead of `math.pi` - it is sometimes frowned upon and considered a bad practice in the Python community. Why so? Because you do not know what members the module defines and you risk either re-defining these definitions in your code, or vice-versa. See the following (bad) example:<jupyter_code>pi = 3.14
from math import *
print(pi)<jupyter_output>3.141592653589793<jupyter_text>Another _feature_ of the `import` statement is the ability to give aliases, or nicknames, to anything we import. Using the syntax `import ... as ...` or `from ... import ... as ...`, we can avoid lengthy library names and save precious keystrokes when we are typing! As in the example above, careful with the names you pick to avoid potential conflicts with your own code.Also remember that, by convention, Python programs should have a maximum of 80-100 characters per line. If you ever login to a cluster and have to open code with `vi`, `nano`, or `emacs`, you will thank you fellow Python coder when they kept their lines short!<jupyter_code>import math as m

print(m == math)  # yes, we can compare modules too!
print(m.pi == math.pi)  # should be true!

from math import sin as sine  # numbers don't sin ...

print(sine(m.pi))<jupyter_output>True
True
1.2246467991473532e-16<jupyter_text>Exercise 1>Your introduction to Python students handed in their final assignments but the program you wrote to automatically grade them deleted all their work – forever! You need to submit their grades and you are feeling really bad about all this. But you might have a solution!>Write a short program that assigns a random passing letter grade to your students. The possible grades are: ‘A’, ‘B’, ‘C’. The Python documentation website ([link](https://docs.python.org/3/)) contains a 'Library Reference' that you should bookmark immediately. This reference contains links to the documentation of all modules included in Python, which range from CSV parsers to cryptography tools.The `random` module contains code to handle randomness, for the lack of a better definition. This module defines functions to sample elements of datasets, produce random distributions of numbers, etc. Let's use it to randomly assign grades to our students. How do we know what function to use? Before continuing on, try figuring that out yourself using the `help` function.<jupyter_code>import random  # Python module to work with random... stuff?

# Let's start by defining some students!
students = ['Joao', 'Linnea', 'Keith', 'Josh', 'Irina', 'Amy']

# Now let's define the grades we can pick from
possible_grades = ['A', 'B', 'C']  # we do not want to fail anyone!

# using a for-loop, we assign a random grade to each student
for name in students:
    grade = random.choice(possible_grades)
    print(name, grade)<jupyter_output><NAME>
Linnea C
Keith C
<NAME>
<NAME>
<NAME><jupyter_text>While Python tries to have one explicit way of doing something, programmers are creative and often find alternative routes. It often pays off to spend a little more time to research if there is a _simpler_ or more direct way of getting what you want. If there are multiple ways, chances are some are optimized for particular use cases.Below, you will find _another_ way to solve our first exercise, using the `random.choices` (note the plural) method. Unlike `random.choice`, which returns __one__ element from the input sequence, `random.choices` allows you to pick k random elements. You can always set this k to 1, and emulate the behaviour of `random.choice`, but as far as performance goes, you will be slightly worse: `random.choices` returns a list, which you will have to access to get your one random element. This all takes some (albeit negligible) time, and when you are doing something thousands or millions of times, that tiny speck of time adds up. So, in short, read the documentation to pick the best method available!<jupyter_code>import random

students = ['Joao', 'Linnea', 'Keith', 'Josh', 'Irina', 'Amy']
possible_grades = ['A', 'B', 'C']

# using a for-loop, we assign a random grade to each student
for name in students:
    grade = random.choices(possible_grades, k=1)
    print(name, grade[0])<jupyter_output>Joao B
Linnea A
Keith B
<NAME>
Irina B
Amy A<jupyter_text>Beyond the Standard LibraryWhile Python is a great general purpose language with an incredible number of built-in modules, the Standard Library does not include code for every single task you would want to perform. However, from very early on, Python programmers wrote all sort of specialized code to do all sorts of niche (or not niche) tasks. These third-party libraries found their home in package repositories that you can access using the `pip` or `conda` commands (provided you have them installed to begin with).For example, to install `pandas` you can do `pip install pandas` or `conda install pandas`. To grasp the breadth of libraries available out there, have a look at [PyPI](https://pypi.org).  Reading Tabular Data into DataFrames IntroductionPandas ([link](https://pandas.pydata.org)) is a third-party library that "aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python."To this end, `pandas` defines two new data structures: the `Series` and the `DataFrame`. These data structures are very fast and memory'efficient, even on large datasets. Importantly, `pandas` includes read and write support for a variety of file formats, including CSV (comma-separated values), Stata, and Excel, allowing you to load data in any of these formats, or interconvert between them.Some more general details about `Series` and `DataFrames`:* The `Series` is a one-dimensional array, meaning it is used to represent several observations of _one_ variable, such as the grades of a group of students. Each entry in this one-dimensional column is labeled numerically by default, starting from 0, but `pandas` allows you to provide custom labels that can be whatever we want (e.g., student names). * The `DataFrame` is a collection of `Series` and as such is used to represent _multiple_ variables. For example, the grades of a group of student across several semesters. There are two dimensions in a `DataFrame`, the first (axis 0) being the _rows_ (vertical) and the second (axis 1) being the _columns_ (horizontal). Operations in `pandas` default to the first axis unless you specify otherwise. Keep this in mind.Generally, both `Series` and `DataFrames` can contain any data type, from decimals to Python objects such as dictionaries or even other `Series` and `DataFrames`. However, and quite logically, some operations (e.g. mean) will only work out of the box for certain data types. `pandas` tries its best to auto-assign data types to your data (per-column in the case of the `DataFrame`) but always best to double-check.Finally, because a `DataFrame` is a collection of `Series`, both data structures share quite a number of common methods, including arithmetic operations or descriptive operations (e.g. `info`) Code Let's then start by importing `pandas`. Often, people use `pd` as a nickname to avoid long lines of code. You will see this "convention" across many tutorials online. If your import fails, make sure you installed `pandas` with `pip` or `conda`.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Let's start simple by reading a CSV file using the `read_csv` function. There are other `read_X` functions available for other file formats, all sharing the same mandatory argument (the name of the input file) and supporting different options.<jupyter_code>data = pd.read_csv('gapminder_gdp_oceania.csv')
print(type(data))<jupyter_output><class 'pandas.core.frame.DataFrame'><jupyter_text>Our `data` variable now contains the data included in the CSV file. Because the input file contains several columns, `pandas` stored its data in a `DataFrame`. You can use the `print` function to see the contents of a `DataFrame`,<jupyter_code>print(data)<jupyter_output>country  gdpPercap_1952  gdpPercap_1957  gdpPercap_1962  \
0    Australia     10039.59564     10949.64959     12217.22686   
1  New Zealand     10556.57566     12247.39532     13175.67800   

   gdpPercap_1967  gdpPercap_1972  gdpPercap_1977  gdpPercap_1982  \
0     14526.12465     16788.62948     18334.19751     19477.00928   
1     14463.91893     16046.03728     16233.71770     17632.41040   

   gdpPercap_1987  gdpPercap_1992  gdpPercap_1997  gdpPercap_2002  \
0     21888.88903     23424.76683     26997.93657     30687.75473   
1     19007.19129     18363.32494     21050.41377     23189.80135   

   gdpPercap_2007  
0     34435.36744  
1     25185.00911<jupyter_text>Alternatively, in Jupyter, you can just write the variable name and execute the cell to show the contents of your `Series` or `DataFrame` in a nicer format.<jupyter_code>data<jupyter_output><empty_output><jupyter_text>The columns in our `DataFrame` contain the name of countries in Oceania and their values of GDP per capita across several decades. The rows are labelled numerically, which is not very useful. If you know a column that you want to use as row labels, you can pass it to the `index_col` argument of the `read_X` function call.<jupyter_code>data = pd.read_csv('gapminder_gdp_oceania.csv', index_col='country')
data<jupyter_output><empty_output><jupyter_text>Much better! Now all the values in your `DataFrame` are simply the GDP per capita values and the row labels are the names of the countries.Let's them have a look at the general structure and contents of our `DataFrame` using the `.info()` method. This method prints a short description of all the columns in the `DataFrame`, the number of entries per column, and the data type (int, float, etc). It also prints the first and last row labels, known as _Indexes_ in `pandas` jargon.<jupyter_code>data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 2 entries, Australia to New Zealand
Data columns (total 12 columns):
gdpPercap_1952    2 non-null float64
gdpPercap_1957    2 non-null float64
gdpPercap_1962    2 non-null float64
gdpPercap_1967    2 non-null float64
gdpPercap_1972    2 non-null float64
gdpPercap_1977    2 non-null float64
gdpPercap_1982    2 non-null float64
gdpPercap_1987    2 non-null float64
gdpPercap_1992    2 non-null float64
gdpPercap_1997    2 non-null float64
gdpPercap_2002    2 non-null float64
gdpPercap_2007    2 non-null float64
dtypes: float64(12)
memory usage: 208.0+ bytes<jupyter_text>To print just the column labels of the dataset, you can use `.columns`, which returns a 'list' (actually a `pandas.Index`).<jupyter_code>data.columns<jupyter_output><empty_output><jupyter_text>Likewise, to print the row labels (also known as indexes), use `.index`:<jupyter_code>data.index<jupyter_output><empty_output><jupyter_text>The `.head()` and `.tail()` methods allow you to peek the first/last few lines of the `DataFrame`, making them particularly useful to get a quick glimpse of the structure of the dataset. This is obviously more useful for large datasets, unlike the one we have that contains only two rows.<jupyter_code>data.head()
data.tail()<jupyter_output><empty_output><jupyter_text>By default, both `.head()` and `.tail()` return the first/last 10 rows of the `DataFrame`, but you can change that with the `n` parameter:<jupyter_code>data.head(n=1)  # prints the first row
data.tail(n=1)  # prints last row<jupyter_output><empty_output><jupyter_text>Switching between rows and columns, i.e., transposing the `DataFrame` is done using the `.T` method. Tranposing is sometimes useful to facilitate access to rows/columns, as we'll see below. Note that this operation does not happen _in place_, meaning that your data is not changed. You will have to assign the result of `data.T` to a variable to store the transposed dataset.<jupyter_code>data.T
data
transposed_data = data.T<jupyter_output><empty_output><jupyter_text>Since plenty of `DataFrame` operations return `DataFrame` objects, you can chain several methods after each other. For instance, to extract only the last 2 columns of a dataset, you can combine `.T.` with `.tail()` and `.T` again to return the final `DataFrame` in the same format:<jupyter_code>data.T.tail(n=2).T<jupyter_output><empty_output><jupyter_text>The last of these descriptive methods is `.describe()`, which prints a number of statistics for all columns of the dataset. By default, this method ignores non-numerical columns, as you will see below.<jupyter_code>data.describe()<jupyter_output><empty_output><jupyter_text>To include statistics of _all_ columns, use the `include` function parameter:<jupyter_code>data = pd.read_csv('gapminder_gdp_oceania.csv')  # re-read the dataset without the index_col option
data.describe(include='all')<jupyter_output><empty_output><jupyter_text>In this case, `pandas` is smart enough to differentiate between column types and apply the proper descriptive statistics appropriately. Wherever the method (e.g. `mean`) would not yield a significant result (e.g. in a `string`-like column), it adds the value `NaN` (not a number). `NaN` is a sort of blank or null value in `pandas` that is usually ignored when doing calculations. Finally, `DataFrames` have `.to_X` methods to save their data to disk in a specific format X. For example, `to_csv()` writes the `DataFrame` to a comma-separated value file:<jupyter_code>data.to_csv('dataset_copy.csv')<jupyter_output><empty_output><jupyter_text>Each _writer_ method has a number of parameters that help control which and how information is written to disk. For example, you can specify a separator between columns when using `to_csv` through the `sep` parameter, replacing the default commas by tabs. Alternatively, you might not want to write the header (column labels) to disk. You can use the `help()` function to figure out which parameters are available for each writer function and how to use them.<jupyter_code>data.to_csv('dataset_copy.tsv', sep='\t', header=False)  # saves a tab-separated file without the header row<jupyter_output><empty_output><jupyter_text>Indexing, Selecting, and Grouping _pandas_ DataFramesNow that we have gone over the basics of `DataFrames`, let's have a look at the indexing and selection features that make `pandas` a very nice data analysis tool. First, load a larger dataset and inspect it with the methods we just learned.<jupyter_code>europe = pd.read_csv('gapminder_gdp_europe.csv', index_col='country')
europe.head()<jupyter_output><empty_output><jupyter_text>There are several ways of accessing values of a `DataFrame`. The first is through numerical indexing, just like regular `list` or `string` objects. Since a `DataFrame` is a two-dimensional array, you can specificy a particular position using two 0-based indexes, separated by a comma. The first index concerns the row (vertical), while the second concerns the column (horizontal).<jupyter_code>print(europe.iloc[0, 0])  # returns the first row/first column
print(europe.iloc[1, 0])  # returns the second row/first column
print(europe.iloc[0, 1])  # returns the first row/second column<jupyter_output>1601.056136
6137.076492
1942.2842440000002<jupyter_text>Like in regular Python indexing, you can also use negative numbers in the `iloc` method:<jupyter_code>europe.iloc[-1, 0]  # returns the last row/first column
europe.tail(n=1)<jupyter_output><empty_output><jupyter_text>You can also use regular slicing notation to extract ranges of values. As expected, the slices do not _include_ the value of the last slice index. Importantly, this type of indexing does not return a single value, as in the previous examples, but new `DataFrames` or `Series`, depending on the dimensionality of the result:<jupyter_code>europe.iloc[0:1, 0:1]  # returns the first row/first column
europe.iloc[0:2, 0:2]  # returns the first two rows and first two columns.
europe.iloc[0, :]  # returns the entire first column as a Series
europe.iloc[999]  # Trying to access a non-existent value returns an IndexError<jupyter_output><empty_output><jupyter_text>However, it is better to access rows/columns based on their labels instead of keeping track of their indices. To do so, `pandas` provides the `.loc` method:<jupyter_code>europe.loc['Albania']  # returns the column 'Albania' as a Series<jupyter_output><empty_output><jupyter_text>The `.loc` method can also receive two labels, for column and row labels.<jupyter_code>europe.loc['Albania', 'gdpPercap_1952']  # returns a single value
europe.loc[:, 'gdpPercap_1952']  # returns all rows in the last column<jupyter_output><empty_output><jupyter_text>Finally, you can also access columns by their labels using the following notations:<jupyter_code>europe['gdpPercap_1952']
europe.gdpPercap_1952<jupyter_output><empty_output><jupyter_text>You can also select ranges of values using slicing notation and row/column labels. In this case, however, and unlike the `.iloc` and regular slicing in Python, the last value _is_ included! Very important difference!<jupyter_code>europe.loc['Albania':'Italy']  # contains Italy!!
europe.loc['Albania':'Italy', 'gdpPercap_1952':'gdpPercap_1962']<jupyter_output><empty_output><jupyter_text>`pandas` is also smart enough to do slicing between non-existent labels. In the following example, there is no country called 'Zimbabwe' nor a column containing the GDP per capita in 1960:<jupyter_code>europe.loc['Albania':'Zimbabwe', 'gdpPercap_1952':'gdpPercap_1960']<jupyter_output><empty_output><jupyter_text>This neat feature works due to string comparison:<jupyter_code>'gdpPercap_1960' > 'gdpPercap_1952'<jupyter_output><empty_output><jupyter_text>You can also specificy which rows/columns to select using lists:<jupyter_code>europe.loc['Albania', ['gdpPercap_1952', 'gdpPercap_2007']]<jupyter_output><empty_output><jupyter_text>Note that in this situation, the slicing notation will not work inside each list:<jupyter_code>europe.loc['Albania', ['gdpPercap_1952':'gdpPercap_1960', 'gdpPercap_2007']]<jupyter_output><empty_output><jupyter_text>Trying to access a non-existing column or row using the `.loc` method also returns an error, but this time a KeyError:<jupyter_code>europe.loc['America']<jupyter_output><empty_output><jupyter_text>Now that we have gone over the basic selection features of `pandas`, let's have a look at some of the built-in analysis methods. Remember that for the majority of built-in functions, the operations run by default over the first axis, so per column.<jupyter_code>europe.max()  # returns the maximum value for each column, in a Series
europe.loc['Albania'].max()  # returns the maximum value for the row 'Albania'
europe.loc['Albania'].idxmax()  # returns the column with the maximum value
europe.loc['Italy'].min()  # returns the minimum value of the column
europe.loc['Italy'].idxmin()  # returns the column with the minimum value
europe.loc['Italy'].sum()  # sums the GDP per capita of Italy across all columns
europe.loc['Italy'].mean()  # returns the average GDP per capita of Italy across all columns
europe.mean()  # calculates the mean GDP per capita, per year, for all countries<jupyter_output><empty_output><jupyter_text>To have these functions operate _per row_, we can use the `axis` parameter:<jupyter_code>europe.mean(axis=1)<jupyter_output><empty_output><jupyter_text>Alternatively, we can use the `.T` method to transpose the rows/columns:<jupyter_code>europe.T.mean()<jupyter_output><empty_output><jupyter_text>`pandas` also supports standard algebraic operations between `DataFrames`. The column and row labels are matched automatically, i.e., mismatched labels (missing rows, missing columns) are assigned `NaN` values. This feature allows you to easily add, subtract, multiply, etc, different datasets and carry out powerful analysis with relatively few commands, as we will see below.<jupyter_code># Let's calculate the difference in GDP per capita between Germany and The Netherlands
europe.loc['Netherlands'] - europe.loc['Germany']<jupyter_output><empty_output><jupyter_text>You can also perform comparisons in a similar way. Comparison operators include the usual `>`, `<`, `!=`, and `==`. The result of these comparison operations, unlike before, are _boolean_ `DataFrames`, with `True`/`False` values per entry in the dataset. These new `DataFrames` are also known as masks, and they can be used to _select_ subsections of the original `DataFrame`. Let's run some examples.<jupyter_code># When is Germany's GDP higher than The Netherlands'?
europe.loc['Germany'] > europe.loc['Netherlands']
# When was a country's GDP per capita higher than 10.000?
europe > 10000
# Use the result of a comparison as a boolean mask
rich_countries = europe > 10000  # assign to variable for readability
europe[rich_countries]<jupyter_output><empty_output><jupyter_text>The resulting `DataFrame` has the same _shape_ (number of columns and rows) as the original one, but has `NaN` values wherever the boolean mask was `False`. Because, as mentioned before, `pandas` ignores `NaN` values in most of its operations (unless told otherwise), this is a particularly powerful way of filtering datasets before calculating certain properties.<jupyter_code># If GDP > 10.000 defines a rich country, what is the average GDP per capita of these countries over time?
europe[rich_countries].mean()
europe[rich_countries].describe()  # pay special attention to the COUNT row.<jupyter_output><empty_output><jupyter_text>Exercise 2> Write code to calculate the fold difference in GDP per capita, per year, between rich and poor countries in Europe, assuming that rich countries are those with a GDP above the mean?<jupyter_code># Create boolean masks
rich_mask = europe > europe.mean()
poor_mask = europe <= europe.mean()

# Filter the dataset
rich_countries = europe[rich_mask]
poor_countries = europe[poor_mask]

# Calculate the fold difference per year
fold_diff = (rich_countries.mean() - poor_countries.mean()) / poor_countries.mean()
fold_diff<jupyter_output><empty_output><jupyter_text>Exercise 3> Write code to calculate how often each country belonged in the 'rich' category. Hint: in Python, 1 and 0 are also synonymous of True and False, respectively.<jupyter_code># In Python, True and False evaluate as integers 1 and 0, respectively
True + True  # should be 2
# We can use the .sum() function on a boolean array to count how many True values there are.
rich_mask.sum(axis=1)<jupyter_output><empty_output><jupyter_text>Exercise 4>Write code to separate European countries in groups based on how often they made it into the 'rich' group, and then calculate the average GDP per capita of each of these groups over time.<jupyter_code>rich_score = rich_mask.sum(axis=1) / len(europe.columns)  # normalize by number of countries
rich_score<jupyter_output><empty_output><jupyter_text>You can use the `.groupby()` function of `pandas` to group rows of a `DataFrame` by a specific metric. This metric should be one-dimensional (e.g. a `Series`), mapping row label (in our case, the country name) to the value you want to group by. The result is __not__ a `DataFrame` but a `DataFrameGroupBy` object, which you can consider a sort of dictionary whose keys are our discrete metric values and the values are row labels that belong to that group.<jupyter_code>grouped_countries = europe.groupby(rich_score)
grouped_countries
grouped_countries.sum()  # some operations propagate through the different groups and return DataFrames
grouped_countries.mean()  # the answer to our exercise question
grouped_countries.groups  # to access the 'groups' individually
# 1.0 = countries that were ALWAYS above the average, in terms of GDP per capita
grouped_countries.get_group(1.0)  # returns a DataFrame with the rows corresponding to this particular group.<jupyter_output><empty_output><jupyter_text>Exercise 5> What do the following lines of code do?>>```first = pd.read_csv('gapminder_all.csv', index_col='country')second = first[first['continent'] == 'Americas']third = second.drop('Puerto Rico')fourth = third.drop('continent', axis=1)```<jupyter_code># Read entire gapminder dataset and read row labels from the country column.
first = pd.read_csv('gapminder_all.csv', index_col='country')
# Produce a new DataFrame that contains only rows where column 'continent' is 'Americas'
second = first[first['continent'] == 'Americas']
# Delete row 'Puerto Rico'
third = second.drop('Puerto Rico')
# Delete column 'continent'
fourth = third.drop('continent', axis=1)<jupyter_output><empty_output><jupyter_text>Creating plots from datasets using matplotlib IntroductionMatplotlib ([link](https://matplotlib.org/)) is a third-party library that produces publication quality figures in a variety of static formats (e.g. PNG, PDF, JPG) and interactive enviroments. The library tries to "make the easy things easy and the hard things possible." In practice, this means that it is very easy to produce simple charts, like line plots or scatter plots, but also possible to change every single bit about them. For example, if you'd like to replace for axis ticks by pictures of french baguettes, you can.While there are several interfaces to use `matplotlib`, we will focus on the `pyplot` module and on the direct integration with `pandas`. Code Jupyter notebook have a few _magic_ commands that enable specific features. For `matplotlib`, it is practical to have plots and figures show directly in the notebook after issuing the commands. The `%matplotlib` allows us to control this (and other) behavior:<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt  # import the pyplot module, the 'easy' interface to matplotlib<jupyter_output><empty_output><jupyter_text>Let's start by plotting a line chart. For simplicity, we will provide the data in two Python lists, corresponding to the x and y axis values:<jupyter_code>time = [0, 1, 2, 3]
position = [0, 100, 200, 300]

plt.plot(time, position)
plt.xlabel('Time (hours)')
plt.ylabel('Position (km)')<jupyter_output><empty_output><jupyter_text>If instead we would like to plot the same data with points instead of a line connecting the data, and color the points red, we can use the `'ro'` format string (syntax and options [here](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.plot.html)):<jupyter_code>plt.plot(time, position, 'ro')
plt.xlabel('Time (hours)')
plt.ylabel('Position (km)')<jupyter_output><empty_output><jupyter_text>Thankfully, `pandas` integrates tightly with `matplotlib`. Although not as flexible as using `matplotlib` itself, this integration makes data visualization very easy and straightforward. Let's give it a try on the Gapminder datasets.<jupyter_code>oceania = pd.read_csv('gapminder_gdp_oceania.csv', index_col='country')
oceania.info()  # quick overview of rows and columns<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 2 entries, Australia to New Zealand
Data columns (total 12 columns):
gdpPercap_1952    2 non-null float64
gdpPercap_1957    2 non-null float64
gdpPercap_1962    2 non-null float64
gdpPercap_1967    2 non-null float64
gdpPercap_1972    2 non-null float64
gdpPercap_1977    2 non-null float64
gdpPercap_1982    2 non-null float64
gdpPercap_1987    2 non-null float64
gdpPercap_1992    2 non-null float64
gdpPercap_1997    2 non-null float64
gdpPercap_2002    2 non-null float64
gdpPercap_2007    2 non-null float64
dtypes: float64(12)
memory usage: 208.0+ bytes<jupyter_text>Let's start by plotting the GDP per capita of Australia over time using the `.plot()` method of `DataFrames`:<jupyter_code>oceania.loc['Australia'].plot()<jupyter_output><empty_output><jupyter_text>Works! But the horizontal (x) axis labels are missing ... why so? By default, `pandas` uses ordinals (first col, second col, etc) as x axis labels. If the indexes happen to be numerical, `pandas` is smart enough to use them as labels. Let's change that then.<jupyter_code>column_labels = oceania.columns  # get column labels
short_labels = column_labels.str.strip('gdpPercap_')  # use string method 'strip' directly on ALL entries of the dataframe
int_labels = short_labels.astype(int)  # cast strings to integers
oceania.columns = int_labels  # replace original labels
oceania
oceania.loc['Australia'].plot()<jupyter_output><empty_output><jupyter_text>Works!!! `pandas` interprets the column names as numbers now and labels the x axis appropriately. Much more informative! What if we want to plot the evolution of the GDP per capita of Australia _and_ New Zealand side by side?<jupyter_code>oceania.plot()<jupyter_output><empty_output><jupyter_text>Doesn't really work as expected ... The column labels (years) are read as data series, instead of the rows (countries). Let's transpose the `DataFrame` and re-plot:<jupyter_code>oceania.T.plot()
plt.ylabel('GDP Per Capita')<jupyter_output><empty_output><jupyter_text>Nicely plotted! Note that we add the y label using `plt.ylabel`, which refers to the `matplotlib.pyplot` interface. How come? `pandas` integrates with `matplotlib` and as such, we can use the `pyplot` interface to customize our plots as we want. In essence, we can define settings in `matplotlib` and these will be used in `pandas` plots. Let's see below how we can customize our plots using this mechanism and the different built-in styles of `matplotlib` (list [here](https://matplotlib.org/3.1.0/gallery/style_sheets/style_sheets_reference.html)):<jupyter_code>plt.style.use('ggplot')
oceania.T.plot()
plt.style.use('seaborn')
oceania.T.plot()
# For fun, there is also a XKCD (from the comics) style.
# However, this is a PERMANENT setting. In order NOT to ruin this Jupyter notebook, we can
# use the 'with' statement of Python that isolates the changes to indented lines (technical term: context manager):
with plt.xkcd():
    oceania.T.plot()<jupyter_output><empty_output><jupyter_text>Let's revert the style to the 'classic' style to continue<jupyter_code>plt.style.use('default')<jupyter_output><empty_output><jupyter_text>`pandas` also offers support to plot other types of charts besides lines. You can choose how to plot your data using the `kind` parameter to `plot()`. Valid options include: `bar`, `barh`, `hist`, `scatter`, `box`, among others. Let's play with a few of these.<jupyter_code>oceania.T.plot(kind='bar')
oceania.T.plot(kind='barh')<jupyter_output><empty_output><jupyter_text>Plotting in Jupyter is nice, but it'd be also useful to save our charts to separate files, in PNG or PDF format, to include in papers, posters, emails, etc. In a regular Python script, we can simply use the `plt.savefig()` method, but in Jupyter ...<jupyter_code>plt.savefig('my_pretty_figure.png')<jupyter_output><empty_output><jupyter_text>0 Axes ... ? If you open the figure (you can switch to the browser tab Jupyter first opened when it launched) you will see that it is just a blank canvas. Our figure was not saved to disk. Why is this? The `.plot()` method of `DataFrames` calls `matplotlib` under the hood and when you call `plt.savefig()`, it requests a new/reset figure canvas.The trick is to assign the result of `plot()` to a variable. This method returns a `matplotlib.Axis` object that has a `get_figure()` method to get access to the figure you see displayed on the screen. Then, you can call `savefig` directly on this figure object.<jupyter_code>ax = oceania.T.plot()
fig = ax.get_figure()
fig.savefig('my_pretty_figure.png')  # this now writes a proper figure to file, go check!<jupyter_output><empty_output><jupyter_text>Final ExercisesNow that we went over the basic of `pandas.DataFrames`, selection, indexing, and plotting using `matplotlib`, we are ready to do some analysis on our datasets. Let's load the combined Gapminder dataset, which includes far more than just GDP per capita values, and have fun!<jupyter_code>gapminder_all = pd.read_csv('gapminder_all.csv', index_col='country')
gapminder_all.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 142 entries, Algeria to New Zealand
Data columns (total 37 columns):
continent         142 non-null object
gdpPercap_1952    142 non-null float64
gdpPercap_1957    142 non-null float64
gdpPercap_1962    142 non-null float64
gdpPercap_1967    142 non-null float64
gdpPercap_1972    142 non-null float64
gdpPercap_1977    142 non-null float64
gdpPercap_1982    142 non-null float64
gdpPercap_1987    142 non-null float64
gdpPercap_1992    142 non-null float64
gdpPercap_1997    142 non-null float64
gdpPercap_2002    142 non-null float64
gdpPercap_2007    142 non-null float64
lifeExp_1952      142 non-null float64
lifeExp_1957      142 non-null float64
lifeExp_1962      142 non-null float64
lifeExp_1967      142 non-null float64
lifeExp_1972      142 non-null float64
lifeExp_1977      142 non-null float64
lifeExp_1982      142 non-null float64
lifeExp_1987      142 non-null float64
lifeExp_1992      142 non-null float64
lifeExp_1997      142 non-nul[...]<jupyter_text>Let's start with plotting the distribution of population size for all countries in the dataset, for 2007.<jupyter_code>pop_2007 = gapminder_all['pop_2007'] / 1e6  # millions
pop_2007.plot(kind='hist')
# Let's zoom in on that first peak (countries with less than 300.000.000 inhabitants)
pop_2007_small = pop_2007[pop_2007 < 300]
pop_2007_small.plot(kind='hist', bins=20)
(pop_2007 < 50).sum() / len(pop_2007.index)  # remember, True = 1<jupyter_output><empty_output><jupyter_text>Interesting. The majority of countries (84.5%) has less than 50 million inhabitants. But how much of the world population lives in these countries?<jupyter_code>pop_2007_tiny = pop_2007[pop_2007 < 50]
pop_2007_tiny.sum() / pop_2007.sum()<jupyter_output><empty_output><jupyter_text>25% !!! 75% of the world population lives in 15% of the countries. Wow! And wealth?<jupyter_code>gdp_2007 = gapminder_all['gdpPercap_2007']
gdp_2007_perc = 100* gdp_2007 / gdp_2007.sum()  # percentage
gdp_2007_perc
gdp_2007_perc.plot(kind='hist')<jupyter_output><empty_output><jupyter_text>Most countries contribute less than 1% of the world's combined GDP per capita. Who are the top 10 richest countries? How much % of the world's combined GDP do they represent?<jupyter_code>gdp_2007_top10 = gdp_2007.sort_values(ascending=False).head(n=10)  # sort by descending value
gdp_2007_top10
100* gdp_2007_top10.sum() / gdp_2007.sum()<jupyter_output><empty_output><jupyter_text>25% !! The 10 richest countries represent a quarter of the world's combined GDP. Let's calculate these values per continent now, to get an idea of the geographical distribution of wealth.<jupyter_code>g = gapminder_all['gdpPercap_2007'].groupby(gapminder_all['continent'])  # group countries by continent
g_sum = g.sum()  # this returns a DataFrame
g_sum.plot(kind='bar')
plt.ylabel('Cumulative GDP per Capita (2007)')<jupyter_output><empty_output><jupyter_text>Europe comes out as the 'winner', followed by Asia. Note that `pandas` automatically colors the several series differently.Now let's check the wealth distribution _within_ each continent using box plots. The `.plot(kind='box')` method will not work properly after `.groupby`, because this last returns a collection of `pandas.Series` objects. However, `pandas` has a `DataFrame.boxplot` method that has a `column` parameter and a `by` parameter, allowing us to plot exactly what we want:<jupyter_code>gapminder_all.boxplot(column=['gdpPercap_2007'], by='continent')
plt.ylabel('GDP per Capita (2007)')<jupyter_output><empty_output><jupyter_text>Europe and Asia have the largest distribution. The American continent has two clear outliers (guess who?) while Oceania (with only two countries in this dataset) is obviously symmetrical. LEt's do the same plot for population size.<jupyter_code>gapminder_all.boxplot(column=['pop_2007'], by='continent')
plt.ylabel('Population Size (2007)')<jupyter_output><empty_output><jupyter_text>China and India dwarf everyone else with their larger than 1 billion populations!! Now let's have a look at another part of the dataset - life expectancy. How did it evolve over time? Does it correlate at all to GDP?<jupyter_code>lifeExp_evol = gapminder_all['lifeExp_2007'] - gapminder_all['lifeExp_1952']
lifeExp_evol.plot(kind='box', xticks=[])  # clear the sole x tick and label by providing an empty list
plt.xlabel('Change in Life Expectancy between 1952 and 2007')<jupyter_output><empty_output><jupyter_text>The median of life expectancy evolution is around 20 years, with some outliers going as far as adding nearly 40 years! However, there are some countries with a _negative_ change, meaning people life _less_ today (in 2007) than they did in 1952. Let's see who these outliers are.<jupyter_code>lifeExp_max = lifeExp_evol.max()
lifeExp_max_country = lifeExp_evol.idxmax()
print(lifeExp_max, lifeExp_max_country)
lifeExp_Oman = gapminder_all.loc['Oman', 'lifeExp_1952':'lifeExp_2007']
lifeExp_Oman.index = lifeExp_Oman.index.str.strip('lifeExp_').astype(int)  # same trick as before, but this is a pandas.Series
lifeExp_Oman.plot()
plt.ylabel('Life Expectancy (years)')
lifeExp_neg = lifeExp_evol[lifeExp_evol < 0]
lifeExp_neg
# We can add Algeria as a 'reference' African country
lifeExp_negCountries = gapminder_all.loc[['Algeria', 'Swaziland', 'Zimbabwe'], 'lifeExp_1952':'lifeExp_2007']  # double indexing!
lifeExp_negCountries.columns = lifeExp_negCountries.columns.str.strip('lifeExp_').astype(int)
lifeExp_negCountries.T.plot()
plt.ylabel('Life Expectancy (years)')<jupyter_output><empty_output><jupyter_text>What happened in the 1990s that justifies this massive drop? A quick Google search 'blames' economical downturn for Zimbabwe and the HIV/AIDS epidemic for Swaziland.Let's make a final plot correlating GDP per Capita with life expectancy. Does a higher living standard correlate with longer lifespans?<jupyter_code>gapminder_all.plot(kind='scatter', x='gdpPercap_2007', y='lifeExp_2007')
plt.xlabel('GDP per Capita (2007)')
plt.ylabel('Life Expectancy (2007)')<jupyter_output><empty_output><jupyter_text>Higher GDP per capita is strongly correlated with a higher life expectancy, while lower GDP per capita is not necessarily associated with a low life expectancy. Or is a matter of scale? There are so many countries in the <10.000 GDP range that is might hard to distinguish a correlation there. Let's try plotting the GDP per Capita on a logarithmic scale.<jupyter_code># Add the log as a new column
# gapminder_all['gdpPercap_2007_log'] = np.log(gapminder_all['gdpPercap_2007'])
gapminder_all.plot(x='gdpPercap_2007', y='lifeExp_2007', logx=True, kind='scatter')<jupyter_output><empty_output><jupyter_text>Correcting for the very large spread in GDP with a logarithmic scale helps visualize the correlation better. We can also obtain a numerical value for the correlation using the `DataFrame.corr()` method. This method returns the pairwise correlations between all columns of the `DataFrame`. Using the `method` parameter, you can select the type of correlation out of `pearson`, `spearman`, and `kendall`.<jupyter_code>gapminder_all.loc[:, ['gdpPercap_2007', 'lifeExp_2007']].corr(method='spearman')  # returns a new DataFrame<jupyter_output><empty_output><jupyter_text>For the final exercise, let's take this plot and color the countries by continent. To do this, we will use a combination of `groupby` and `matplotlib` to plot multiple series in the same chart. For simplicity, we will also add a new column to the dataset to include the log values of the GDP per capita in 2007. This exercise wraps up several topics we covered throughout the lesson and showcases the potential of combining `pandas` and `matplotlib` to create high-quality figures from raw data.<jupyter_code>import math  # to access the log function

gdp_2007_log = gapminder_all['gdpPercap_2007'].apply(math.log)  # applies a function on the entire series
gapminder_all['gdpPercap_2007_log'] = gdp_2007_log  # adds a new column

# Use the 'flexible' matplotlib interface
fig, ax = plt.subplots()  # create figure with one subplot (default)

# Create mapping of continent to colors
colors = {'Africa': 'darkred', 
          'Europe': 'skyblue', 
          'Americas': 'orange',
          'Asia': 'lightgreen',
          'Oceania': 'magenta'}

# Use groupby to separate DataFrame by continent
groups = gapminder_all.groupby(gapminder_all['continent'])

for name, g in groups:  # iterate over groups
    ax.scatter(g['gdpPercap_2007_log'], g['lifeExp_2007'],  # x/y data
               c=colors[name],  # color of this particular series
               s=g['pop_2007']/1e6,  # scale the marker by the population size (divided by 1 million)
              alpha=0.75  # transparency
              )

ax.set_xlabel('GDP per Capita (2007) (log)')
ax.set_ylabel('Life Expectancy (2007)')

fig.savefig('world_lifeExp_vs_GDP.png')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07996294502810704, 'fraction_numerical': 0.027770174958418426, 'mean_word_length': 5.279481755684823, 'pattern_counts': {'":': 0, '<': 318, '<?xml version=': 0, '>': 327, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '78', 'dir': 'jupyter-structured-clean-dedup', 'id': '436228', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 15154, 'n_tokens_neox': 13779, 'n_words': 5826, 'path': 'python-gapminder-july-2019/SIEPR_part_II.ipynb', 'repo_name': 'JoaoRodrigues/sc-teaching', 'size': '47501'}