<jupyter_start><jupyter_text>Project: GDP/capita Investigation  Data: obtained from GapMinder.org/data - GDP per capita US inflation adjusted for 2010. Additional data includes Gini coefficient, & Corruption perception index. Data Descriptions:1. GDP per capita is gross domestic product divided by midyear population. GDP is the sum of gross value added by all resident **producers** in the economy. 1. Gini shows income inequality in a society. A higher number means more inequality. 1. CPI (Corruption Perception Index) is transparency international's score of perceptions of corruption. Higher values indicate LESS corruption)   Questions:1.  How does the world's countries GDP/capita compare with one another?After calculating the **Compound Annual Growth Rate (CAGR)** for all country's GDP/capita from 2000-2018, determine characteristics such as:2. Which regions, as determined by the World Bank, contain the top 10 fastest growing countries?3. Which income groups are represented, as determined by the World Bank, by the top 10 fastest growing countries (highest CAGR values)?4. What did the GDP/capita trend look like for the top 10 countries? See Society Data notebook1. How did perceived corruption change, if any, during expansive growth?2. How did the Gini coefficient change, if any, during expansive growth? Table of ContentsData WranglingExploratory Data Analysis            Question 1        Question 2        Question 3        Question 4        Conclusions  Environment & Datasets Setup Utilizing an economic growth dataset, **GDP/capita (US$ inflation adjusted)** from gapminder.com/data<jupyter_code>import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats

sns.set_style('darkgrid')

#mpl.style.use('ggplot')
%matplotlib inline

### Remove scientific notation
#pd.set_option('display.float_format', lambda x: '%1.0f' % x)

### Read Total GDP by PPP adjusted for inflation
gdp = pd.read_csv("data_income/gdppercapita_us_inflation_adjusted.csv")
gdp.head()<jupyter_output><empty_output><jupyter_text>Data Wrangling Load country dataset which provides World Bank regions & income groupsBack to top<jupyter_code># get countries dataframe to have ability to divide world into regions or income groups
# drop columns not needed, & rename for merge and ease of use

country_df = pd.read_csv("country_regions.csv")
country_df.rename(columns={'name':'country', 'World bank, 4 income groups 2017':'2017 income groups'}, inplace=True)
country_df.drop(['geo','World bank, 3 income groups 2017','members_oecd_g77','Latitude','Longitude','UN member since'],axis=1,inplace=True)
country_df.head()
# Merge the GDP/capita & country dataset into ONE dataset

gdp20up = pd.merge(gdp, country_df, on='country')
gdp20up.head(1)
# merge GDP/capita & country datasets

gdp2 = pd.merge(gdp, country_df, on="country")
gdp2.head(3)<jupyter_output><empty_output><jupyter_text>Missing Data Determine year range and evaluate missing data according to that rangeBack to top<jupyter_code>gdp2.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 190 entries, 0 to 189
Data columns (total 65 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   country             190 non-null    object 
 1   1960                90 non-null     float64
 2   1961                91 non-null     float64
 3   1962                91 non-null     float64
 4   1963                91 non-null     float64
 5   1964                91 non-null     float64
 6   1965                95 non-null     float64
 7   1966                98 non-null     float64
 8   1967                99 non-null     float64
 9   1968                101 non-null    float64
 10  1969                101 non-null    float64
 11  1970                112 non-null    float64
 12  1971                112 non-null    float64
 13  1972                112 non-null    float64
 14  1973                112 non-null    float64
 15  1974                114 non-null    float64
 16  1975    [...]<jupyter_text>Too many missing values from 1960 - 1999. Will utilize years 2000-2018 for data analysis<jupyter_code># Create list of years that will be dropped from the dataset, years 1960-1999. These years have too many missing values.

years_drop20 = [str(x) for x in range(1960,2000,1)]
# drop years & verify
gdp20up.drop(years_drop20, axis=1, inplace=True)
gdp20up.head(1)<jupyter_output><empty_output><jupyter_text>How many missing values per row/country, from 2000 - 2018?I'm certain there's a much easier way to do this. However, I have resorted to creating a series of missing values by index. Added country's names back, and filtered out the values greater than 0.<jupyter_code>missing_vals20 = gdp20up.isnull().sum(axis=1)
missing_vals20
# create series of countries name to add back to missing_values
countries_list20 = gdp20up.iloc[:,0]
# create dictionary of countries & missing values, create dataframe, filter out 0 missing values, & sort greatest to least
frame20 = {'country': countries_list20, 'missing values': missing_vals20}
country_nulls20 = pd.DataFrame(frame20)
country_nulls20 = country_nulls20[country_nulls20['missing values'] > 0]
country_nulls20.sort_values('missing values', axis=0, ascending=False, inplace=True, na_position='last')
country_nulls20<jupyter_output><empty_output><jupyter_text>Drop countries whose missing values exceed 18Decided to drop 5 countries according to missing values above. Reviewing the data, it was determined the countries would not be in the running for "fastest growing country", 7+ values is too many to fill with alternative data<jupyter_code># drop countries according to index
gdp20up.drop([47,96,155,54,118], inplace=True)
# Review where missing values occur in data set for remaining countries by utilizing the index from 
# dataframe above, country_nulls20

gdp20up.loc[[185,0,77,142,143],:]<jupyter_output><empty_output><jupyter_text>Forward fill missing values<jupyter_code># make a copy in case you need to jump back
gdp20up3 = gdp20up.copy()
gdp20up3.loc[[185,0,77,142,143],:]
# forward fill & verify
gdp20up3 = gdp20up3.ffill(axis='columns')
gdp20up3.loc[[185,0,77,142,143],:]
# Manually forward fill remaining entries 

gdp20up3.loc[0,'2001'] = 330
gdp20up3.loc[0,'2000'] = 330
gdp20up3.loc[143,'2000'] = 844<jupyter_output><empty_output><jupyter_text>Verify if missing values are gone<jupyter_code>gdp20up3.apply(lambda row: pd.isnull(row).sum(), axis=1).sum()<jupyter_output><empty_output><jupyter_text>Value types<jupyter_code># GDP/capita has turned into an 'object'. Not sure how this occurred. Perhaps with the forward fill.
gdp20up3.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 185 entries, 0 to 189
Data columns (total 25 columns):
 #   Column              Non-Null Count  Dtype 
---  ------              --------------  ----- 
 0   country             185 non-null    object
 1   2000                185 non-null    object
 2   2001                185 non-null    object
 3   2002                185 non-null    object
 4   2003                185 non-null    object
 5   2004                185 non-null    object
 6   2005                185 non-null    object
 7   2006                185 non-null    object
 8   2007                185 non-null    object
 9   2008                185 non-null    object
 10  2009                185 non-null    object
 11  2010                185 non-null    object
 12  2011                185 non-null    object
 13  2012                185 non-null    object
 14  2013                185 non-null    object
 15  2014                185 non-null    object
 16  2015                185 no[...]<jupyter_text>Data Wrangling Process Finished!  Exploratory Data Analysis begins Evaluate GDP/capita distributionBack to top<jupyter_code># Need to focus on just the years columns, not regions & income groups, or country names
year2000up = [str(x) for x in range(2000,2019,1)]
year2000up
# created DF for investigating years only as float. Without reconstructing the dataframe

gdps_only = gdp20up3[year2000up].apply(pd.to_numeric)
gdps_only
gdps_only.to_csv('gdps_only.csv',index=False)
gdps_only.describe()
gdps2000 = gdps_only['2000']
gdps2005 = gdps_only['2005']
gdps2010 = gdps_only['2010']
gdps2015 = gdps_only['2015']
gdps2018 = gdps_only['2018']
gdps2000_outliers = gdp20up3.loc[gdp20up3['2000'] > 80000]
gdps2000_outliers
gdp20up3.loc[gdp20up3['country'] == 'United States']<jupyter_output><empty_output><jupyter_text>1 How does the world's countries GDP/capita compare with one another?Back to top<jupyter_code>plt.figure(figsize=(10,10))
sns.distplot(gdps2000, bins=20, kde=False, rug=True);

plt.xlabel('GDP per capita', fontsize=20)
plt.title('Year 2000 - GDP/capita distribution for all countries', fontsize=24)
plt.ylabel('Number of Countries', fontsize=20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12);<jupyter_output><empty_output><jupyter_text>The visualization indicates that a majority of countries have a GDP/capita  80000, are Monaco, Luxembuorg, & Norway, all of whom are Western European.  Determine fastest growing countries of the world Calculate & Add the Compound Annual Growth Rate (CAGR) column<jupyter_code># calculate exponent value to add to calc_CAGR. 1/x where x = the number of years to calculate
# 2000-2018 = 19 years
1/19
# define function that returns the CAGR when called

def calc_CAGR(row):
    return ((((row['2018'] / row['2001']) ** 0.052632) - 1) * 100)
# apply function to every row in the dataset & check results

gdp20up3['CAGR'] = gdp20up3.apply(lambda row: calc_CAGR(row),axis=1)
gdp20up3.head(3)
# Sort dataframe by Compound Annual Growth Rate (CAGR), greatest to least
gdp20up3.sort_values('CAGR',axis=0, ascending=False,inplace=True,na_position='last')

#gdp5.iloc[0:15,-12:]
#gdp5.iloc[0:15,columns_to_show]
gdp20up3.head(10)
# create dataframe of top 10 exclusively
CAGR_top10 = gdp20up3.head(10)<jupyter_output><empty_output><jupyter_text>2: Which regions, as determined by the World Bank, contain the top 10 fastest growing countries according to GDP/capita CAGR, 2000-2018?Back to top<jupyter_code>gdp20up3.eight_regions.unique()
eight_regionsCAGR = CAGR_top10.groupby('eight_regions')['CAGR'].mean()
eight_regionsCAGR
other2 = pd.Series([0,0,0,0], index=['america_north','america_south','europe_west','africa_north'], name='Other Regions')
other2
top10_CAGR = eight_regionsCAGR.append(other2)
top10_CAGR
# Compare the 8 regions where the top 10 fastest growing economies fall within, averaged

locations = [1,2,3,4,5,6,7,8]
heights = top10_CAGR
plt.figure(figsize=(14,8))

# labels = ['African Sub Saharan','Asia West','East Asia Pacific','Europe East','America North','America South','Europe West','Africa North']
labels = top10_CAGR.index.str.replace('_',' ').str.title() # alternative to line above

#eight_regionsCAGR.plt(kind='bar',tick_label=labels)
plt.bar(locations, heights, tick_label=labels)

plt.title('Average CAGR for World Bank Regions\n of the top 10 fastest growing countries', fontsize=26)
plt.xlabel('Regions', fontsize=20)
plt.ylabel('Average CAGR (%)', fontsize=20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12);<jupyter_output><empty_output><jupyter_text>The coutries with the greatest annulaized GPP/capita from 2000-2018 are categorized as Sub Saharan African, Western Asian, Eastern Pacific Asian, & East European. Eastern Pacific Asian has the greatest average CAGR.  3. Which income groups are represented, as determined by the World Bank, by the top 10 fastest growing countries?Back to top<jupyter_code>CAGR_top10
income_groupsCAGR = CAGR_top10.groupby('2017 income groups')['CAGR'].mean()
income_groupsCAGR
# get 2017 income group names
gdp20up3['2017 income groups'].unique()
# add high income group to income group series
income_group = pd.Series([0],index=['high income'],name='income group')
income_group
income_groupsCAGR = income_groupsCAGR.append(income_group)
income_groupsCAGR
# Compare the 4 income groups where the top 10 fastest growing economies fall within, averaged

locations = [1,2,3,4]
heights = income_groupsCAGR
plt.figure(figsize=(10,6))

# labels = ['African Sub Saharan','Asia West','East Asia Pacific','Europe East','America North','America South','Europe West','Africa North']
#.str.replace('_',' ').str.title() # alternative to line above
labels = income_groupsCAGR.index


#eight_regionsCAGR.plt(kind='bar',tick_label=labels)
plt.bar(locations, heights, tick_label=labels)

plt.title('Average CAGR for World Bank Income Groups\n of top 10 fastest growing countries', fontsize=24)
plt.xlabel('Income Groups', fontsize=20)
plt.ylabel('Average CAGR (%)', fontsize=20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12);<jupyter_output><empty_output><jupyter_text>The coutries with the greatest annulaized GPP/capita from 2000-2018 are categorized Low income, Lower Middle income, & Upper middle income. The upper middle income group has the greatest mean CAGR AND is Eastern Pacific Asian.  4 What did the GDP/capita trend look like for the top 10 countries?To perform these plots, the dataframe will need to be transposed such that the years are in a column, and the countries are in a row.Back to top<jupyter_code># Create copy of top 10 CAGR dataframe to restructure for line plot of GDPs
CAGR_top10re = CAGR_top10.copy()
CAGR_top10re.head(3)
# remove excess columns & check result
CAGR_top10re.drop(['four_regions','six_regions','eight_regions','World bank region','2017 income groups'], axis=1, inplace=True)
CAGR_top10re.head(3)
# Univariate Data Exploration:
# Explore the dispersion of the data
CAGR_top10re
# make copy before transposing, just in case
CAGR_top10re2 = CAGR_top10re.copy()
# perform the transpose
CAGR_top10re2 = CAGR_top10re2.transpose()
CAGR_top10re2.head(3)
# rename all column names with the first row of the dataframe with vectorizing

CAGR_top10re2.columns = CAGR_top10re2.iloc[0]
# the index needs fixing after transposing
CAGR_top10re2 = CAGR_top10re2.reset_index()
CAGR_top10re2.head(3)
# remove CAGR & country row and check result
# rename "index" column to "year"
CAGR_top10re2.drop(0,inplace=True)
CAGR_top10re2.drop(20,inplace=True)
CAGR_top10re2 = CAGR_top10re2.rename(columns={'index':'year'})
CAGR_top10re2.head(3)
# rename "index" column to "year"
CAGR_top10re2 = CAGR_top10re2.rename(columns={'index':'year'})
CAGR_top10re2.head(3)<jupyter_output><empty_output><jupyter_text>Create line plot of the Top 5 fastest growing countries from 2000 - 2018<jupyter_code>ax = plt.gca()

CAGR_top10re2.plot(x='year', y='Myanmar', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='China', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='Azerbaijan', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='Turkmenistan', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='Armenia', kind='line', figsize=(14,8), ax=ax)

plt.title("GDP/capita Trend for the Top 5 fastest growing countries from 2000-2018", fontsize=24)
plt.ylabel("GDP/capita ($)", fontsize=20)
plt.xlabel("Years", fontsize=20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12);<jupyter_output><empty_output><jupyter_text>The line chart shows the degree of which the top 5 fastest growing countries grew. One can see that Armenia & Azerbaijan was affected by the global financial crisis, of 2007-2008, caused by the greed of the wealthy. One can see that Myanmar, China, & Turkmenistan were relatively unscathed by the global crisis.  Create line plot of the Top 6-10 fastest growing countries from 2000 - 2018<jupyter_code>ax = plt.gca()

CAGR_top10re2.plot(x='year', y='Ethiopia', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='Georgia', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='Cambodia', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='Bhutan', kind='line', figsize=(14,8), ax=ax)
CAGR_top10re2.plot(x='year', y='Mongolia', kind='line', figsize=(14,8), ax=ax)

plt.title("GDP/capita Trend for the Top 6-10 fastest growing countries from 2000-2018", fontsize=24)
plt.ylabel("GDP/capita ($)", fontsize=20)
plt.xlabel("Years", fontsize=20)
plt.xticks(fontsize=12)
plt.yticks(fontsize=12);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10373596711858168, 'fraction_numerical': 0.06803263603459911, 'mean_word_length': 4.582876712328767, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '557274', 'license': 'MIT', 'n_tokens_mistral': 6053, 'n_tokens_neox': 5101, 'n_words': 1616, 'path': 'Investigate_a_Dataset_pg.1.ipynb', 'repo_name': 'ozihandicraft13/InvestigateData', 'size': '16301'}