<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project 1: World Progress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, you'll explore data from [Gapminder.org](http://gapminder.org), a website dedicated to providing a fact-based view of the world and how it has changed. That site includes several data visualizations and presentations, but also publishes the raw data that we will use in this project to recreate and extend some of their most famous visualizations.\n",
    "\n",
    "The Gapminder website collects data from many sources and compiles them into tables that describe many countries around the world. All of the data they aggregate are published in the [Systema Globalis](https://github.com/open-numbers/ddf--gapminder--systema_globalis/blob/master/README.md). Their goal is \"to compile all public statistics; Social, Economic and Environmental; into a comparable total dataset.\" All data sets in this project are copied directly from the Systema Globalis without any changes.\n",
    "\n",
    "This project is dedicated to [<NAME>ling](https://en.wikipedia.org/wiki/Hans_Rosling) (1948-2017), who championed the use of data to understand and prioritize global development challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get started, load `datascience`, `numpy`, and `plots`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datascience import *\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plots\n",
    "plots.style.use('fivethirtyeight')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "## 1. Global Population Growth\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "The global population of humans reached 1 billion around 1800, 3 billion around 1960, and 7 billion around 2011. The potential impact of exponential population growth has concerned scientists, economists, and politicians alike.\n",
    "\n",
    "The UN Population Division estimates that the world population will likely continue to grow throughout the 21st century, but at a slower rate, perhaps reaching 11 billion by 2100. However, the UN does not rule out scenarios of more extreme growth.\n",
    "\n",
    "<a href=\"http://www.pewresearch.org/fact-tank/2015/06/08/scientists-more-worried-than-public-about-worlds-growing-population/ft_15-06-04_popcount/\"> \n",
    " <img src=\"pew_population_projection.png\"/> \n",
    "</a>\n",
    "\n",
    "In this section, we will examine some of the factors that influence population growth and how they are changing around the world.\n",
    "\n",
    "The first table we will consider is the total population of each country over time. Run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "population = Table.read_table('population.csv').where(\"time\", are.below(2021))\n",
    "population.show(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** The population csv file can also be found [here](https://github.com/open-numbers/ddf--gapminder--systema_globalis/raw/master/ddf--datapoints--population_total--by--geo--time.csv). The data for this project was downloaded in April 2020."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Haiti\n",
    "\n",
    "In the `population` table, the `geo` column contains three-letter codes established by the [International Organization for Standardization](https://en.wikipedia.org/wiki/International_Organization_for_Standardization) (ISO) in the [Alpha-3](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-3#Current_codes) standard. We will begin by taking a close look at Haiti. Use the ISO link to find the 3-letter code for Haiti."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 1.** Create a table called `h_pop` that has two columns labeled `time` and `population_total`. The first column should contain the years from 1970 through 2020 (including both 1970 and 2020) and the second should contain the population of Haiti in each of those years.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "h_pop = ...\n",
    "h_pop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false
   },
   "source": [
    "Run the following cell to create a table called `h_decade` that has the population of Haiti for every ten years starting in 1970 and going up to 2020. At a glance, it appears that the population of Haiti has been growing quickly indeed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "h_pop.set_format('population_total', NumberFormatter)\n",
    "\n",
    "tens = np.arange(1970, 2021, 10) # 1970, 1980, 1990, ...\n",
    "h_decade = h_pop.sort('time').where('time', are.contained_in(tens))\n",
    "h_decade.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 2.** Assign `initial` to an array that contains the population for every ten year interval from 1970 to 2010. Then, assign `changed` to an array that contains the population for every ten year interval from 1980 to 2020. You should use the `h_decade` table to create both arrays, first filtering the table to only contain the relevant years.\n",
    "\n",
    "The annual growth rate for a time period is equal to:\n",
    "\n",
    "$$(\\frac{\\text{Population at end of period}}{\\text{Population at start of period}}^{\\frac{1}{\\text{number of years}}} )-1$$\n",
    "\n",
    "We have provided the code below that uses  `initial` and `changed` in order to add a column to `h_decade` called `annual_growth`. Don't worry about the calculation of the growth rates; run the test below to test your solution.\n",
    "\n",
    "If you are interested in how we came up with the formula for growth rates, consult the [growth rates](https://www.inferentialthinking.com/chapters/03/2/1/growth) section of the textbook.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial = ...\n",
    "changed = ...\n",
    "\n",
    "growth_rates = ((changed/initial)**0.1)-1\n",
    "\n",
    "h_1970_through_2010 = h_decade.where('time', are.below_or_equal_to(2010)) \n",
    "h_decade_growth = h_1970_through_2010.with_column('annual_growth', growth_rates)\n",
    "# Don't change this line!\n",
    "h_decade_growth.set_format('annual_growth', PercentFormatter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false
   },
   "source": [
    "While the population has grown every decade since 1970, the annual growth rate decreased dramatically from 1980 to 2020. Let's look at some other information in order to develop a possible explanation. Run the next cell to load three additional tables of measurements about countries over time."
   ]
  },
  {<commit_msg>add solutions<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project 1: World Progress"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, you'll explore data from [Gapminder.org](http://gapminder.org), a website dedicated to providing a fact-based view of the world and how it has changed. That site includes several data visualizations and presentations, but also publishes the raw data that we will use in this project to recreate and extend some of their most famous visualizations.\n",
    "\n",
    "The Gapminder website collects data from many sources and compiles them into tables that describe many countries around the world. All of the data they aggregate are published in the [Systema Globalis](https://github.com/open-numbers/ddf--gapminder--systema_globalis/blob/master/README.md). Their goal is \"to compile all public statistics; Social, Economic and Environmental; into a comparable total dataset.\" All data sets in this project are copied directly from the Systema Globalis without any changes.\n",
    "\n",
    "This project is dedicated to [<NAME>](https://en.wikipedia.org/wiki/Hans_Rosling) (1948-2017), who championed the use of data to understand and prioritize global development challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get started, load `datascience`, `numpy`, and `plots`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datascience import *\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plots\n",
    "plots.style.use('fivethirtyeight')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "## 1. Global Population Growth\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "The global population of humans reached 1 billion around 1800, 3 billion around 1960, and 7 billion around 2011. The potential impact of exponential population growth has concerned scientists, economists, and politicians alike.\n",
    "\n",
    "The UN Population Division estimates that the world population will likely continue to grow throughout the 21st century, but at a slower rate, perhaps reaching 11 billion by 2100. However, the UN does not rule out scenarios of more extreme growth.\n",
    "\n",
    "<a href=\"http://www.pewresearch.org/fact-tank/2015/06/08/scientists-more-worried-than-public-about-worlds-growing-population/ft_15-06-04_popcount/\"> \n",
    " <img src=\"pew_population_projection.png\"/> \n",
    "</a>\n",
    "\n",
    "In this section, we will examine some of the factors that influence population growth and how they are changing around the world.\n",
    "\n",
    "The first table we will consider is the total population of each country over time. Run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "population = Table.read_table('population.csv').where(\"time\", are.below(2021))\n",
    "population.show(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** The population csv file can also be found [here](https://github.com/open-numbers/ddf--gapminder--systema_globalis/raw/master/ddf--datapoints--population_total--by--geo--time.csv). The data for this project was downloaded in April 2020."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Haiti\n",
    "\n",
    "In the `population` table, the `geo` column contains three-letter codes established by the [International Organization for Standardization](https://en.wikipedia.org/wiki/International_Organization_for_Standardization) (ISO) in the [Alpha-3](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-3#Current_codes) standard. We will begin by taking a close look at Haiti. The three-letter code for Haiti is 'hti'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "**Question 1.** Create a table called `h_pop` that has two columns labeled `time` and `population_total`. The first column should contain the years from 1970 through 2020 (including both 1970 and 2020) and the second should contain the population of Haiti in each of those years.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1_1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "h_pop = ...\n",
    "h_pop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false
   },
   "source": [
    "Run the following cell to create a table called `h_decade` that has the population of Haiti for every ten years starting in 1970 and going up to 2020. At a glance, it appears that the population of Haiti has been growing quickly indeed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false
   },
   "outputs": [],
   "source": [
    "h_pop.set_format('population_total', NumberFormatter)\n",
    "\n",
    "tens = np.arange(1970, 2021, 10) # 1970, 1980, 1990, ...\n",
    "h_decade = h_pop.sort('time').where('time', are.contained_in(tens))\n",
    "h_decade.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false
   },
   "source": [
    "**Question 2.** Assign `initial` to an array that contains the population for every ten year interval from 1970 to 2010. Then, assign `changed` to an array that contains the population for every ten year interval from 1980 to 2020. You should use the `h_decade` table to create both arrays, first filtering the table to only contain the relevant years.\n",
    "\n",
    "The annual growth rate for a time period is equal to:\n",
    "\n",
    "$$(\\frac{\\text{Population at end of period}}{\\text{Population at start of period}}^{\\frac{1}{\\text{number of years}}} )-1$$\n",
    "\n",
    "We have provided the code below that uses  `initial` and `changed` in order to add a column to `h_decade` called `annual_growth`. Don't worry about the calculation of the growth rates; run the test below to test your solution.\n",
    "\n",
    "If you are interested in how we came up with the formula for growth rates, about how yearly exponential growth would play out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "initial = ...\n",
    "changed = ...\n",
    "\n",
    "growth_rates = ((changed/initial)**0.1)-1\n",
    "\n",
    "h_1970_through_2010 = h_decade.where('time', are.below_or_equal_to(2010)) \n",
    "h_decade_growth = h_1970_through_2010.with_column('annual_growth', growth_rates)\n",
    "# Don't change this line!\n",
    "h_decade_growth.set_format('annual_growth', PercentFormatter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false
   },
   "source": [
    "While the population has grown every decade since 1970, the annual growth rate decreased dramatically from 1980 to 2020. Let's look at some other information in order to develop a possible explanation. Run the next cell to load three additional tables of measurements about countries over time."
   ]
  },
  {
{'fraction_non_alphanumeric': 0.1435044339439446, 'fraction_numerical': 0.02194316784257881, 'mean_word_length': 3.2809642560266004, 'pattern_counts': {'":': 148, '<': 14, '<?xml version=': 0, '>': 14, 'https://': 11, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5933191', 'n_tokens_mistral': 5198, 'n_tokens_neox': 4735, 'n_words': 1614}