<jupyter_start><jupyter_text>_Lambda School Data Science_ Sequence your narrativeToday we will create a sequence of visualizations inspired by [<NAME>'s 200 Countries, 200 Years, 4 Minutes](https://www.youtube.com/watch?v=jbkSRLYSojo).Using this [data from Gapminder](https://github.com/open-numbers/ddf--gapminder--systema_globalis/):- [Income Per Person (GDP Per Capital, Inflation Adjusted) by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv)- [Life Expectancy (in Years) by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv)- [Population Totals, by Geo & Time](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv)- [Entities](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv)- [Concepts](https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv) Objectives- sequence multiple visualizations- combine qualitative anecdotes with quantitative aggregatesLinks- [<NAME>’s TED talks](https://www.ted.com/speakers/hans_rosling)- [Spiralling global temperatures from 1850-2016](https://twitter.com/ed_hawkins/status/729753441459945474)- "[The Pudding](https://pudding.cool/) explains ideas debated in culture with visual essays."- [A Data Point Walks Into a Bar](https://lisacharlotterost.github.io/2016/12/27/datapoint-in-bar/): a thoughtful blog post about emotion and empathy in data storytelling  Make a plan How to present the data?Variables --> Visual Encodings- Income --> x- Lifespan --> y- Region --> color- Population --> size- Year --> animation frame (alternative: small multiple)- Country --> annotationQualitative --> Verbal- Editorial / contextual explanation --> audio narration (alternative: text) How to structure the data?| Year | Country | Region   | Income | Lifespan | Population ||------|---------|----------|--------|----------|------------|| 1818 | USA     | Americas |     |        |           || 1918 | USA     | Americas |    |       |          || 2018 | USA     | Americas |   |       |         || 1818 | China   | Asia     |       |         |           || 1918 | China   | Asia     |      |        |         || 2018 | China   | Asia     |     |       |       |  Upgrade SeabornMake sure you have at least version 0.9.0.In Colab, go to **Restart runtime** after you run the `pip` command.<jupyter_code>!pip install --upgrade seaborn
import seaborn as sns
sns.__version__<jupyter_output><empty_output><jupyter_text>More imports<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>Load & look at data<jupyter_code>income = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--income_per_person_gdppercapita_ppp_inflation_adjusted--by--geo--time.csv')
lifespan = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--life_expectancy_years--by--geo--time.csv')
population = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--datapoints--population_total--by--geo--time.csv')
entities = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv')
concepts = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--concepts.csv')
income.shape, lifespan.shape, population.shape, entities.shape, concepts.shape
income.head()
lifespan.head()
population.head()
pd.options.display.max_columns = 500
entities.head()
concepts.head()<jupyter_output><empty_output><jupyter_text>Merge data https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf<jupyter_code>df = pd.merge(lifespan, income)
df = pd.merge(df, population)
df.shape
df = pd.merge(df, entities[["country", "name", "world_6region"]],
             how="inner", left_on="geo", right_on="country")
df.head()
df = df.drop(columns=["geo", "country"])
df = df.rename(columns={
    "time": "year", 
    "income_per_person_gdppercapita_ppp_inflation_adjusted": "income", 
    "life_expectancy_years": "lifespan", 
    "population_total": "population", 
    "name": "country", 
    "world_6region": "region"})

df.tail()<jupyter_output><empty_output><jupyter_text>Explore data<jupyter_code>df.describe()
df.describe(exclude=np.number)
df.country.unique()
usa = df[df.country == "United States"]
usa[usa.year.isin([1818, 1918, 2018])]
china = df[df.country == "China"]
china[china.year.isin([1818, 1918, 2018])]<jupyter_output><empty_output><jupyter_text>Plot visualization<jupyter_code>now = df[df.year == 2018]
now.shape
now.hist();
pd.scatter_matrix(now);
sns.relplot(x="income", 
            y="lifespan", 
            hue="region", 
            size="population", 
            data=now);<jupyter_output><empty_output><jupyter_text>Analyze outliers<jupyter_code>now[now.income > 80000].sort_values(by="income")
qatar = now[now.country == "Qatar"]
qatar_income = qatar.income.values[0]
qatar_lifespan = qatar.lifespan.values[0]
sns.relplot(x="income", y="lifespan", hue="region", size="population", data=now);

plt.text(x=qatar_income - 5000, y=qatar_lifespan + 1, s="Qatar")
plt.title("Qatar has the highest income in 2018");<jupyter_output><empty_output><jupyter_text>Plot multiple years<jupyter_code>years = [1818, 1918, 2018]

centuries = df[df.year.isin(years)]

sns.relplot(x="income", 
            y="lifespan", 
            hue="region", 
            size="population", 
            col="year", 
            data=centuries)

plt.xscale("log");<jupyter_output><empty_output><jupyter_text>Point out a story<jupyter_code>years = [1918, 1938, 1958, 1979, 1998, 2018]

for year in years:
  sns.relplot(x="income", 
              y="lifespan", 
              hue="region", 
              size="population", 
              data=df[df.year==year])
              
  plt.xscale("log")
  plt.xlim((150, 1500000))
  plt.ylim((20, 90))
  plt.title(year)
  plt.axhline(y=50, color="grey");<jupyter_output><empty_output><jupyter_text>ASSIGNMENTReplicate the lesson code STRETCH OPTIONS 1. Animate!- [Making animations work in Google Colaboratory](https://medium.com/lambda-school-machine-learning/making-animations-work-in-google-colaboratory-new-home-for-ml-prototyping-c6147186ae75)- [How to Create Animated Graphs in Python](https://towardsdatascience.com/how-to-create-animated-graphs-in-python-bb619cc2dec1)- [The Ultimate Day of Chicago Bikeshare](https://chrisluedtke.github.io/divvy-data.html) (Lambda School Data Science student) 2. Work on anything related to your portfolio site / project<jupyter_code># Importing a dataset from the U.S. Energy Information Administration
# to begin cleaning/sorting data for project.

# Hoping to begin working on a plot for this data.
# Would like to try a stacked area chart and use animation to show how it 
# changes over time.
from google.colab import files
uploaded = files.upload()
import io
df = pd.read_csv(io.BytesIO(uploaded['MER_T07_02A.csv']))

df.head()

df.shape
# drop first column because we have that info elsewhere

df = df.drop(columns=["MSN"])
df.shape
df["YYYYMM"].tail(20)
 # only going to keep the rows with 13th month because it is total for that year
  
 # change dtype to string so I can filter using endswith()
  
df["YYYYMM"] = df["YYYYMM"].astype('str')
# filter for values ending in "13"

df = df[df["YYYYMM"].str.endswith("13")]
df.tail()
df.shape
# change dtype back to int so i can do math

df["YYYYMM"] = df["YYYYMM"].astype("int64")
df["Year"] = (df["YYYYMM"] - 13) / 100
df.head()
#drop YYYYMM column and reorder remaining columns

df = df.drop(columns=["YYYYMM"])
df = df[['Year', 'Value', 'Column_Order', 'Description', 'Unit']]
df.head()
# check out Unit column

df["Unit"].describe()
# remove column and remember to label any plots with 
# "Million Kilowatthours"

df = df.drop(columns=["Unit"])
# check out description column

df["Description"].unique()
# pivot the data to create more columns

table = df.pivot(index="Year", columns="Description", values="Value")
table
# clean up column names

table = table.rename(columns={"Electricity Net Generation From Coal, All Sectors": "Coal",
                     "Electricity Net Generation From Conventional Hydroelectric Power, All Sectors": "Hydroelectric Power",
                     "Electricity Net Generation From Geothermal, All Sectors": "Geothermal",
                     "Electricity Net Generation From Hydroelectric Pumped Storage, All Sectors": "Hydroelectric Pumped Storage",
                     "Electricity Net Generation From Natural Gas, All Sectors": "Natural Gas",
                     "Electricity Net Generation From Nuclear Electric Power, All Sectors": "Nuclear Electric Power",
                     "Electricity Net Generation From Other Gases, All Sectors": "Other Gases",
                     "Electricity Net Generation From Petroleum, All Sectors": "Petroleum",
                     "Electricity Net Generation From Solar, All Sectors": "Solar",
                     "Electricity Net Generation From Waste, All Sectors": "Waste",
                     "Electricity Net Generation From Wind, All Sectors": "Wind",
                     "Electricity Net Generation From Wood, All Sectors": "Wood",
                     "Electricity Net Generation Total (including from sources not shown), All Sectors": "Total"})
table.head()
table = table.reset_index()
table.head()
table["Year"].isnull().sum()
table["Year"].describe()
table["Coal"].isnull().sum()
table["Coal"].describe()
table["Coal"] = table["Coal"].astype("float64")
table["Coal"].describe()
plt.scatter(x=table["Year"], y=table["Coal"]);


df3 = pd.DataFrame(np.array(([1,892,3], [4,895,6], [7, 872, 9])), 
                   index=['mouse', 'rabbit', 'duck'], 
                   columns=['one', 'two', 'three'])
df3
df3.filter(regex='e$', axis=1)
df3.filter(like='bbi', axis=0)
df3["two"] = df3["two"].astype('str')
df3["two"].dtype
df4 = df3["two"].str.endswith("2")
df4
df4 = df3["two"].filter(regex="2$")
df4<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1504905229069435, 'fraction_numerical': 0.02266882560243833, 'mean_word_length': 3.9833887043189367, 'pattern_counts': {'":': 19, '<': 43, '<?xml version=': 0, '>': 53, 'https://': 20, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '99069', 'license': 'MIT', 'n_tokens_mistral': 3740, 'n_tokens_neox': 3352, 'n_words': 737, 'path': 'module4-sequence-your-narrative/LS_DS_224_Sequence_your_narrative.ipynb', 'repo_name': 'livjab/DS-Unit-1-Sprint-3-Data-Storytelling', 'size': '10511'}