<jupyter_start><jupyter_text>_Lambda School Data Science_ Sequence your narrativeToday we will create a sequence of visualizations inspired by [<NAME>'s 200 Countries, 200 Years, 4 Minutes](https://www.youtube.com/watch?v=jbkSRLYSojo).Using this [data from Gapminder](https://github.com/open-numbers/ddf--gapminder--systema_globalis/):- [Income Per Person (GDP Per Capital, Inflation Adjusted) by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv)- [Life Expectancy (in Years) by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv)- [Population Totals, by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv)- [Entities](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv)- [Concepts](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv) Objectives- sequence multiple visualizations- combine qualitative anecdotes with quantitative aggregatesLinks- [<NAME>’s TED talks](https://www.ted.com/speakers/hans_rosling)- [Spiralling global temperatures from 1850-2016](https://twitter.com/ed_hawkins/status/729753441459945474)- "[The Pudding](https://pudding.cool/) explains ideas debated in culture with visual essays."- [A Data Point Walks Into a Bar](https://lisacharlotterost.github.io/2016/12/27/datapoint-in-bar/): a thoughtful blog post about emotion and empathy in data storytelling  Make a plan How to present the data?Variables --> Visual Encodings- Income --> x- Lifespan --> y- Region --> color- Population --> size- Year --> animation frame (alternative: small multiple)- Country --> annotationQualitative --> Verbal- Editorial / contextual explanation --> audio narration (alternative: text) How to structure the data?| Year | Country | Region   | Income | Lifespan | Population ||------|---------|----------|--------|----------|------------|| 1818 | USA     | Americas |     |        |           || 1918 | USA     | Americas |    |       |          || 2018 | USA     | Americas |   |       |         || 1818 | China   | Asia     |       |         |           || 1918 | China   | Asia     |      |        |         || 2018 | China   | Asia     |     |       |       |  Upgrade SeabornMake sure you have at least version 0.9.0.In Colab, go to **Restart runtime** after you run the `pip` command.<jupyter_code>#!pip install --upgrade seaborn
import seaborn as sns
sns.__version__<jupyter_output><empty_output><jupyter_text>More imports<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
def start():
    options = {
        'display': {
            'max_columns': None,
            'max_colwidth': 25,
            'expand_frame_repr': False,  # Don't wrap to multiple pages
            'max_rows': 14,
            'max_seq_items': 50,         # Max length of printed sequence
            'precision': 4,
            'show_dimensions': False
        },
        'mode': {
            'chained_assignment': None   # Controls SettingWithCopyWarning
        }
    }

    for category, option in options.items():
        for op, value in option.items():
            pd.set_option(f'{category}.{op}', value)  # Python 3.6+

if __name__ == '__main__':
    start()
    del start  # Clean up namespace in the interpreter<jupyter_output><empty_output><jupyter_text>Load & look at data<jupyter_code>income = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv')
lifespan = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv')
population = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv')
entities = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv')
concepts = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv')
income.shape, lifespan.shape, population.shape, entities.shape, concepts.shape
income.head()
lifespan.head()
population.head()
pd.options.display.max_columns = 500
entities.head()
concepts.head()
csv_dict = {
    'income': 'https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv',
    'lifespan': 'https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv',
    'population': 'https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv',
    'entities': 'https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv',
    'concepts': 'https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv'
}

dfs = {}

for k in csv_dict:
  dfs[k] = pd.read_csv(csv_dict[k])
dfs.keys()<jupyter_output><empty_output><jupyter_text>Merge data https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf<jupyter_code>dfs['income'].head()
dfs['lifespan'].head()
dfs['population'].head()
dfs['entities'].head().T
df = pd.merge(dfs['income'], dfs['lifespan'])
df = pd.merge(df, dfs['population'])
df_v2 = dfs['income'].merge(dfs['lifespan']).merge(dfs['population'])
#df = pd.merge(dfs['income'], dfs['lifespan'])
df_v2.sample(10)
df.equals(df_v2)
dfs['entities'][['country', 'name', 'world_6region']].sample(5)
df = pd.merge(df, dfs['entities'][['country', 'name', 'world_6region']], left_on='geo', right_on='country')
df.head()
df = df.drop(columns=['geo', 'country'])
df.head()
col_name_map = {
    'time': 'year',
    'income_per_person_gdppercapita_ppp_inflation_adjusted': 'income',
    'life_expectancy_years': 'lifespan',
    'population_total': 'population',
    'name': 'country',
    'world_6region': 'region'
}
df = df.rename(columns = col_name_map)
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>Explore data<jupyter_code>df.info()
df.dtypes
df.describe()
df[df['lifespan'] == 1]
df.describe(exclude=np.number)
df['country'].unique()
df['country'].nunique()
df[df['year'] >= 2000].describe()
usa = df[df['country'] == "United States"]
usa[usa['year'].isin([1818, 1918, 2018])]
usa['year'].isin([1850, 1950])
china = df[df['country'] == "China"]
china[china['year'].isin([1818, 1918, 2018])]<jupyter_output><empty_output><jupyter_text>Plot visualization<jupyter_code>now = df[df['year'] == 2018]
now.shape
now.sample(10)
sns.distplot(np.log10(now['population']))
sns.relplot(x='income', y='lifespan', hue='region', size='population', data=now)<jupyter_output><empty_output><jupyter_text>Analyze outliers<jupyter_code>now[now['income'] > 80000].sort_values('income')
now[now['income'] < 1000].sort_values('income')
now[now['lifespan'] < 60].sort_values('lifespan')
def plot_annotate_country(country_name, text_ann):
  country = now[now['country'] == country_name]
  country_income = country['income'].values[0]
  country_lifespan = country['lifespan'].values[0]
  sns.relplot(x='income', y='lifespan', hue='region', size='population', data=now)
  plt.text(country_income-5000, country_lifespan+1, country_name)
  plt.title(f'{country_name} {text_ann}')
country_name = 'Somalia'
text_ann = 'is one of the richest countries'
plot_annotate_country(country_name, text_ann)<jupyter_output><empty_output><jupyter_text>Plot multiple years<jupyter_code>sns.relplot(x='income',
            y='lifespan',
            hue='region',
            size='population',
            data=df[df['year'].isin(range(1818, 2019, 50))],
            row='year')<jupyter_output><empty_output><jupyter_text>Point out a story<jupyter_code>sns.relplot(x='income', y='lifespan', hue='region', size='population', data=now)
plt.axhline(y=60, color='grey')
plt.axvline(x=60000, color='grey')
year = 1937 #@param {type: "slider", min: 1800, max: 2018, step: 1}

sns.relplot(x='income', y='lifespan', hue='region', size='population', data=df[df.year == year])

plt.xscale('log')
plt.xlim((150, 150000))
plt.ylim((20, 90))
plt.title(year)
from google.colab import widgets

years = [1819, 1919, 2000]
tb = widgets.TabBar([str(year) for year in years])
for tab, year in zip(tb, years):
  sns.relplot(x='income', y='lifespan', hue='region', size='population', data=df[df.year==year])
  
  plt.xscale('log')
  plt.xlim((150, 150000))
  plt.ylim((20, 90))<jupyter_output><empty_output><jupyter_text>ASSIGNMENTReplicate the lesson code STRETCH OPTIONS 1. Animate!- [Making animations work in Google Colaboratory](https://medium.com/lambda-school-machine-learning/making-animations-work-in-google-colaboratory-new-home-for-ml-prototyping-c6147186ae75)- [How to Create Animated Graphs in Python](https://towardsdatascience.com/how-to-create-animated-graphs-in-python-bb619cc2dec1)- [The Ultimate Day of Chicago Bikeshare](https://chrisluedtke.github.io/divvy-data.html) (Lambda School Data Science student) 2. Work on anything related to your portfolio site / project<jupyter_code>new_csv_dict={
    "income": "https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv",
    "lifespan": "https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv",
    "population": "https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv",
    "entities": "https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv",
    "concepts": "https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv"
}
new_csv_dict.keys()
new_dfs={}
for k in new_csv_dict:
  new_dfs[k] = pd.read_csv(new_csv_dict[k])

new_dfs.keys()
new_df = pd.merge(new_dfs['income'], new_dfs['lifespan'])
new_df = pd.merge(new_df, new_dfs['population'])
new_df.head()
new_df_v2 = dfs['income'].merge(dfs['lifespan']).merge(dfs['population'])
new_df_v2.head()
new_dfs['entities'][['country', 'name', 'world_6region']]
new_df = pd.merge(new_df, new_dfs['entities'][['country', 'name', 'world_6region']], left_on="geo", right_on="country")
new_df.head()
new_df = new_df.drop(columns=["geo", "country"])
new_df.head()
new_col_name_map = {
    'time': 'year',
    'income_per_person_gdppercapita_ppp_inflation_adjusted': 'income',
    'life_expectancy_years': 'lifespan',
    'population_total': 'population',
    'name': 'country',
    'world_6region': 'region'
}

new_df = new_df.rename(columns=new_col_name_map);
new_df.head()
new_df.info()
new_df.dtypes
new_df.describe()
new_df[new_df['lifespan'] == 1]
new_df.describe(exclude=np.number)
new_df['country'].unique()
new_df['country'].nunique()
usa = new_df[new_df['country'] == "United States"]
usa[usa['year'].isin([1818, 1918, 2018])]
zambia = new_df[new_df['country'] == "Zambia"]
zambia[zambia['year'].isin([1818, 1918, 2018])]
new_now = new_df[new_df['year'] == 2018]
new_now.head()
sns.distplot(np.log10(new_now['population']))
sns.relplot(x='income', y='lifespan', hue='region', size='population', data=new_now)
country_name = 'China'
text_ann = 'is one of the richest countries'
plot_annotate_country(country_name, text_ann)
sns.relplot(x='income',
            y='lifespan',
            hue='region',
            size='population',
            data=df[df['year'].isin(range(1800, 2000, 50))],
            row='year')
new_year = 1984 #@param {type: "slider", min: 1800, max: 2018, step: 1}

sns.relplot(x='income', y='lifespan', hue='region', size='population', data=df[df.year == year])

plt.xscale('log')
plt.xlim((150, 150000))
plt.ylim((20, 90))
plt.title(year)
new_years = [1800, 1900, 2000]
new_tb = widgets.TabBar([str(year) for year in years])
for tab, year in zip(new_tb, new_years):
  sns.relplot(x='income', y='lifespan', hue='region', size='population', data=df[df.year==year])
  
  plt.xscale('log')
  plt.xlim((150, 150000))
  plt.ylim((20, 90))
import matplotlib.animation as animation<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17506424733276224, 'fraction_numerical': 0.02593255976948836, 'mean_word_length': 4.850569476082004, 'pattern_counts': {'":': 5, '<': 45, '<?xml version=': 0, '>': 54, 'https://': 30, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '510695', 'license': 'MIT', 'n_tokens_mistral': 4982, 'n_tokens_neox': 4509, 'n_words': 706, 'path': 'module4-sequence-your-narrative/Ashwin_Raghav_Swamy_LS_DS_224_Sequence_your_narrative.ipynb', 'repo_name': 'ash12hub/DS-Unit-1-Sprint-2-Data-Wrangling-and-Storytelling', 'size': '12853'}