<jupyter_start><jupyter_text><NAME>  Research question/interestsIdentify what indicators can be used to clearly visualize inequality.In other words, identify which indicators are better indicators of inequality or which indicators are highly correlated (to the Corruption Perception Index).<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np 
import seaborn as sns

import datapackage
import pandas as pd

data_url = 'https://datahub.io/core/corruption-perceptions-index/datapackage.json'

package = datapackage.Package(data_url)

resources = package.resources
for resource in resources:
    if resource.tabular:
        data = pd.read_csv(resource.descriptor['path'])
        print (data)
WID = pd.read_csv('../data/raw/WID_countries.csv', sep = ";")
WID
WID.head()
WID.tail()
WID.head(8)<jupyter_output><empty_output><jupyter_text>By using head() function, it is easy to find the columns.<jupyter_code># I want to take all african countries "xpi_xxx.csv" into one csv file.
# use "WID_countries.csv" and "AfricanDevelopment.csv"

WID = (pd.read_csv('../data/raw/WID_countries.csv', sep = ";"))

African = (pd.read_csv('../data/raw/AfricanDevelopment.csv'))

# Only take out African countries.
WID_African = WID.query('region == "Africa"') # df.query('state == "CA"')

# Change the column name of WID_African (from "titlename" to "country")
WID_African = WID_African.rename(columns={'titlename':'country'})

pd.merge(African, WID_African, on='country', how='outer')

merged_af = pd.merge(African, WID_African, on='country', how='outer')

# sort by "alpha2"
merged_af.sort_values(by='alpha2') 

# Remove lines where "alpha2" is nan
merged_af_dropNan = merged_af.dropna(how='all').dropna()
merged_af_dropNan.sort_values(by='alpha2') 
merged_af_dropNan
merged_af_dropNan["region2"]
merged_af_dropNan.groupby('region2').count()
grouped_by_area = merged_af_dropNan.groupby('region2')<jupyter_output><empty_output><jupyter_text>The number of data is extremely small in South Africa.On the contrary, East and West Africa have a relatively large amount of data.<jupyter_code>grouped_by_area.mean()<jupyter_output><empty_output><jupyter_text>If we average the human development index and the population in 2021 for each region within Africa, we can see that the human development index is not proportional to the man-made population.<jupyter_code>grouped_by_area.max()
grouped_by_area.min()
grouped_by_area.plot(x='humanDevelopmentIndex', y='pop2021')
# for index, row in df.iterrows():
import os

X = merged_af_dropNan.loc[1, 'country'] #    BB = merged_af_dropNan.loc[i, 'country']
x = X.lower()
path = (f'../data/raw/cpi_{x}.csv')
is_file = os.path.isfile(path)

if is_file:
    XXX = (pd.read_csv(f'../data/raw/cpi_{x}.csv', sep = ";"))
    XXX['country']=f"{x}"
    merged_af_dropNan_merged_with_CPI = pd.merge(merged_af_dropNan,XXX)
else:
    pass
XXX
merged_af_dropNan.describe()
merged_af_dropNan.info()
x = len(merged_af_dropNan.country.unique())
print(f"Number of Countries: {x}")
merged_af_dropNan.hist(bins=10, figsize=(20,15), color = 'teal')
from pandas.plotting import scatter_matrix
x = scatter_matrix(merged_af_dropNan, alpha=1, figsize=(20, 10), diagonal='hist')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1285935085007728, 'fraction_numerical': 0.00865533230293663, 'mean_word_length': 6.127753303964758, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '530747', 'license': 'MIT', 'n_tokens_mistral': 1185, 'n_tokens_neox': 1106, 'n_words': 265, 'path': 'notebooks/ungraded/analysis1/submitted_analysis1.ipynb', 'repo_name': 'data301-2021-winter1/solo-sei-wiese', 'size': '3240'}