<jupyter_start><jupyter_text>Gapminder If you come here without expecting Japanese, please click [Google translated version](https://translate.google.com/translate?hl=&sl=ja&tl=en&u=https%3A%2F%2Fpy4etrics.github.io%2FGapminder.html) in English or the language of your choice.---<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from gapminder import gapminder
from see import see<jupyter_output><empty_output><jupyter_text>[Gapminder](https://www.gapminder.org)とは世界規模で見た経済格差をデータで探る有名なサイトであり、一見の価値があるサイトである。そのサイトで使われているデータを整理してパッケージにまとめたのが`gapminder`である。````{note}MacではTerminal、WindowsではGit Bashを使い、次のコマンドで`gapminder`をインストールできる。```pip install gapminder```````ここでは`gapminder`に含まれるデータを使い`pandas`の`groupby`という`DataFrame`のメソッドと`Multi-index`（階層型インデックス）の使い方の例を紹介する。両方ともデータをグループ化して扱う場合に非常に重宝するので、覚えておいて損はしないだろう。  データ<jupyter_code>df = gapminder
df.head()
df.tail()
df.info()
df.describe()<jupyter_output><empty_output><jupyter_text>**含まれる国名**<jupyter_code>countries = df.loc[:,'country'].unique()
countries<jupyter_output><empty_output><jupyter_text>**国数**<jupyter_code>len(countries)<jupyter_output><empty_output><jupyter_text>**`continent`の内訳**<jupyter_code># continentのリスト

continent_list = np.sort(df.loc[:,'continent'].unique()).tolist()
continent_list
# 国数を入れる空のリスト
num_countries = []

for i in continent_list:
    # 条件
    con = df.loc[:,'continent'] == i
    #　条件が満たされる行を抽出
    series_countries = df.loc[con,'country']
    # 重複をなくす
    series_countries_unique = series_countries.unique()
    # 国数を数える
    num = len(series_countries_unique)
    # 上のリストに追加
    num_countries.append(num)
for name, num in zip(continent_list,num_countries):
    print(f'{name}: {num}')
sum(num_countries)<jupyter_output><empty_output><jupyter_text>groupby()<jupyter_code>df_group = df.groupby('continent')
see(df_group)<jupyter_output><empty_output><jupyter_text>continentの内訳（again）<jupyter_code># Seriesを返し，要素はarray

country_names = df_group['country'].unique()
country_names
N = len(country_names)

for i in range(N):
    t0 = country_names[i]
    t1 = len(t0)
    print(country_names.index[i],':',t1)<jupyter_output>Africa : 52
Americas : 25
Asia : 33
Europe : 30
Oceania : 2<jupyter_text>統計量<jupyter_code>three_vars=['lifeExp','pop','gdpPercap']<jupyter_output><empty_output><jupyter_text>データ数<jupyter_code>df_group.count()
df_group.size().plot(kind='bar')
pass<jupyter_output><empty_output><jupyter_text>平均<jupyter_code>df_group[three_vars].mean()
df_group.mean().plot(kind='scatter', x='gdpPercap', y='lifeExp')
pass<jupyter_output><empty_output><jupyter_text>標準偏差<jupyter_code>df_group[three_vars].std()<jupyter_output><empty_output><jupyter_text>最大値<jupyter_code>df_group.max()<jupyter_output><empty_output><jupyter_text>最小値<jupyter_code>df_group.min()<jupyter_output><empty_output><jupyter_text>**基本的統計**<jupyter_code>df_group[three_vars].describe()<jupyter_output><empty_output><jupyter_text>groupby.agg() `agg()`を使うとよりメソッドだけではなく，他の関数も使える。`()`の中に関数を入れる。<jupyter_code>df_group.agg(np.mean)
df_group[three_vars].agg([np.max, np.min, np.mean])
# 自作の関数もOK

func = lambda x : (np.max(x)-np.min(x))/np.mean(x)

df_group['lifeExp','pop','gdpPercap'].agg(func)<jupyter_output><ipython-input-27-11f62f4671e9>:5: FutureWarning: Indexing with multiple keys (implicitly converted to a tuple of keys) will be deprecated, use a list instead.
  df_group['lifeExp','pop','gdpPercap'].agg(func)<jupyter_text>図 **`continent`平均**<jupyter_code>df_lifeExp_continent = df_group['lifeExp'].mean()
df_lifeExp_continent.plot(kind='bar')
pass<jupyter_output><empty_output><jupyter_text>クロス・セクション<jupyter_code>df_mean = df_group.mean()
df_mean['ln_pop'] = np.log(df_mean['pop'])
df_mean['ln_gdpPercap'] = np.log(df_mean['gdpPercap'])
df_mean['lifeExp_10'] = df_mean['lifeExp']/10
df_mean[['lifeExp_10', 'ln_gdpPercap']].plot(kind='bar')
pass<jupyter_output><empty_output><jupyter_text>複数階層の`groupby()` `continent`別の平均時系列を考えるときに有用。<jupyter_code>df_group2 = df.groupby(['continent','year'])
df_group2.mean().head()
# lifeExpの列だけを選択した後，行はyear列はcontinentになるDataFrameに変換

df_lifeExp_group = df_group2.mean().loc[:,'lifeExp'].unstack(level=0)
df_lifeExp_group.plot()
plt.ylabel('Average Life Expectancy')
pass<jupyter_output><empty_output><jupyter_text>**世界平均との比較**<jupyter_code>df_group_year = df.groupby('year')
world_lifeExp = df_group_year.mean()['lifeExp'].values.reshape(1,12).T
df_lifeExp_diff = df_lifeExp_group - world_lifeExp
df_lifeExp_diff.plot()
pass<jupyter_output><empty_output><jupyter_text>Multi-index<jupyter_code># sort_index()がないとWarningがでる場合がある(順番で並ぶとPythonが変数を探しやすくなる)

dfm = df.set_index(['continent','country','year']).sort_index()
dfm.head()<jupyter_output><empty_output><jupyter_text>`continent`, `country`, `year`の３つがインデックス！  統計量  データ数<jupyter_code>dfm.count(level=0)<jupyter_output><empty_output><jupyter_text>平均の計算<jupyter_code>dfm.mean(level='continent')

# dfm.mean(level=0) も同じ<jupyter_output><empty_output><jupyter_text>標準偏差<jupyter_code>dfm.std(level='continent')<jupyter_output><empty_output><jupyter_text>最大値・最小値<jupyter_code>dfm.max(level='continent')
dfm.min(level='continent')<jupyter_output><empty_output><jupyter_text>図 **図（５カ国の時系列）**<jupyter_code>dfm_2 = dfm.droplevel(level=0,axis=0)
countries = ['Japan', 'United Kingdom', 'United States', 'China', 'Thailand']

dfm_2.loc[(countries),'lifeExp'].unstack(level=0).plot()
pass<jupyter_output><empty_output><jupyter_text>**`lifeExp`の世界平均との差**<jupyter_code>df_lifeExp_mi = pd.DataFrame()

for i in continent_list:
    temp = dfm.loc[(i,),'lifeExp'].unstack(level=0).mean(axis=1)
    df_lifeExp_mi[i] = temp
df_lifeExp_mi.plot()
plt.title('Average Life Expectancy')
pass
world_lifeExp_mi = dfm_2['lifeExp'].unstack(level=0).mean(axis=1).values.reshape(12,1)
df_lifeExp_diff_mi = df_lifeExp_mi - world_lifeExp_mi
df_lifeExp_diff_mi.plot()
pass<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1582365003417635, 'fraction_numerical': 0.010252904989747095, 'mean_word_length': 11.587096774193549, 'pattern_counts': {'":': 0, '<': 104, '<?xml version=': 0, '>': 104, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '553807', 'license': 'MIT', 'n_tokens_mistral': 2763, 'n_tokens_neox': 2699, 'n_words': 223, 'path': 'ipynb/Gapminder.ipynb', 'repo_name': 'jungsee/Teaching', 'size': '5852'}