<jupyter_start><jupyter_text>Project: Life Expectancy Data Analysis Table of ContentsIntroductionData WranglingExploratory Data AnalysisLimitationsConclusions  Introduction> I decided to investigate Gapminder World data in order to see if there is a correlation overtime to life expectancies.  There were a number of different variables I could have chosen from but decided to go with: child mortality, co2 emissions, human development, democracy score, and income per person.  With these, I hope to find which one may have the closest correlation to life expectancy over the years.  >  The original data was broken out by country and even though I could investigate life expectancy at a country level, I decided to take the average of all the countries to find the global average life expectancy by year.  I also went ahead and applied the average of all the countries for all the variables.  This helped me start with data that was more clean and could be analyzed at a macro level.<jupyter_code># imports
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Data Wrangling> In this section of the report, I take a closer look at the data to review the years and variables available for each.  This will help me narrow down the data to be as clean as possible. General Properties<jupyter_code># reviewing datasets by loading with pandas
life_expectancy = pd.read_csv('Data/life_expectancy_years.csv')
income_per_person = pd.read_csv('Data/income_per_person_gdppercapita_ppp_inflation_adjusted.csv')
human_development_index = pd.read_csv('Data/hdi_human_development_index.csv')
democracy_score = pd.read_csv('Data/democracy_score_use_as_color.csv')
corruption_preception_index = pd.read_csv('Data/corruption_perception_index_cpi.csv')
co2_emissions = pd.read_csv('Data/co2_emissions_tonnes_per_person.csv')
child_mortality = pd.read_csv('Data/child_mortality_0_5_year_olds_dying_per_1000_born.csv')
# Getting name of dataframe
def get_df_name(df):
    name =[x for x in globals() if globals()[x] is df][0]
    return name

# Running function to clean all dataframees
def global_average_of_metrics(df):
    # getting df name
    name = get_df_name(df)
    # removing country
    new_df = df.drop(columns=['country'])
    # transposing the data so year can be the new index
    new_df = new_df.transpose()
    # creating a new column that takes the average life expectancy across all countries
    new_df[name] = new_df.mean(numeric_only=True, axis=1)
    # removing all other rows except new average column
    new_df = new_df.loc[:, new_df.columns.intersection([name])]
    return new_df
life_expectancy = global_average_of_metrics(life_expectancy)
income_per_person = global_average_of_metrics(income_per_person)
human_development_index = global_average_of_metrics(human_development_index)
democracy_score = global_average_of_metrics(democracy_score)
corruption_preception_index = global_average_of_metrics(corruption_preception_index)
co2_emissions = global_average_of_metrics(co2_emissions)
child_mortality = global_average_of_metrics(child_mortality)<jupyter_output><empty_output><jupyter_text>Analysis of Life Expectancy Dataframe<jupyter_code># initial review
life_expectancy.head()
# reviewing data to see statistical relevance of data
life_expectancy.describe()<jupyter_output><empty_output><jupyter_text>For life expectancy, we see the minimum life expectancy as 22 and maximum at 72.  With 50% being at 34 and the mean being 43, we can see there is not much skewness in the data.   Reformatting Income Per Person DataframeGross domestic product per person adjusted for differences in purchasing power (in international dollars, fixed 2011 prices)<jupyter_code># initial review
income_per_person.head()
# reviewing statistical relevance of data
income_per_person.describe()<jupyter_output><empty_output><jupyter_text>For income per person, we see a min of 947 and max of 29039.  50% of people have an income of 2142 whil the top 75% has an income of 11709.  This is a very large range meaning there are more people who are on the lower end of income.   Reformatting Human Development Index Dataframe<jupyter_code># initial review
human_development_index.head()
# reviewing statistical relevance
human_development_index.describe()<jupyter_output><empty_output><jupyter_text>For human development, the min is 0.5 while the max is 0.6.  The data is very close to each other and this may be due to the limited amount of data available on human development.  I will review further the available years to see the distribution.   Reformatting Democracy Score Dataframe<jupyter_code># initial review
democracy_score.head()
# reviewing statistical relevance
democracy_score.describe()<jupyter_output><empty_output><jupyter_text>The data is very interesting here.  The min for democracy score is -7 while the max is 4.  The mean is -2 meaning majority of countries have a negative democracy score.  It will be interesting to see what impact this may have on life expectancy  Reformatting Corruption Prediction Index Dataframe Transaprency International's score of perceptions of corruption. Higher values indicated less corruption<jupyter_code># initial review
corruption_preception_index.head()
corruption_preception_index.describe()<jupyter_output><empty_output><jupyter_text>The min and max of this data set are similar meaning, there may not be much data available for corruption preception and I will investigate futher the available years for cpi.   Reformatting CO2 Emissions Dataframe Carbon dioxide emissions from the burning of fossil fuel (metric tonnes of CO2 per person).<jupyter_code># initial review
co2_emissions.head()
co2_emissions.describe()<jupyter_output><empty_output><jupyter_text>The min co2 emissions is 0.2 and the max co2 emissions is 5.5.  With 50% percentile being 1.7 this could mean there was a sharp increase in co2 emissions in the later years  Reformatting Child Motality Dataframe Death of children under five years of age per 1,000 living births<jupyter_code># initial review
child_mortality.head()
child_mortality.describe()<jupyter_output><empty_output><jupyter_text>There seems to be a higher rate of child mortality if 50% percentile are at 365.  Min is 29 while mean is 290.  There was probably higher child mortality rates in eariler years.   Combining the datasets<jupyter_code># combining all dataframes
gapminder_data = pd.concat([life_expectancy, income_per_person, human_development_index, democracy_score, corruption_preception_index, co2_emissions, child_mortality], axis=1, join='outer')
# reviewing data
gapminder_data
# checking null values
gapminder_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 241 entries, 1800 to 2040
Data columns (total 7 columns):
life_expectancy                219 non-null float64
income_per_person              241 non-null float64
human_development_index        26 non-null float64
democracy_score                212 non-null float64
corruption_preception_index    6 non-null float64
co2_emissions                  215 non-null float64
child_mortality                219 non-null float64
dtypes: float64(7)
memory usage: 15.1+ KB<jupyter_text>There are too many null values in human development index along with corruption prediction index.  We will go ahead and drop those columns as there is not enough data available to make a good analysis.<jupyter_code># dropping hdi and cpi
gapminder_data.drop(columns=['human_development_index','corruption_preception_index'], axis=1, inplace=True)
# re-reviewing nulls
gapminder_data.info()
# reviewing rows that have nulls
gapminder_data[gapminder_data.isnull().any(axis=1)]<jupyter_output><empty_output><jupyter_text>The rows that have nulls seem to be more recent years along values that are predictions.  I will drop these rows.<jupyter_code># dropping rows that have NaN
gapminder_data = gapminder_data.dropna()
# reviewing nulls
gapminder_data.info()
# reviewing data types 
gapminder_data.dtypes
# resetting index in order to have year as a column
gapminder_data = gapminder_data.reset_index()
gapminder_data
# renaming new column created for year to year
gapminder_data.rename(columns={"index": "year"},inplace=True)
# reviewing data type for year
gapminder_data.dtypes
# changing year column to year dtype
gapminder_data['year'] = pd.DatetimeIndex(gapminder_data['year']).year
gapminder_data.dtypes
gapminder_data<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis> Now that I have clean data, I will go ahead and begin my analysis with an understanding that there will all data available for all variables across all the years.  Research Question 1: Review each variable, what do the independent variables look like?<jupyter_code># Correlation between life expectancy and different metrics
gapminder_data.hist(figsize=(10,8));<jupyter_output><empty_output><jupyter_text>We see that there is a high number of years that had high child mortality rates.  This is heavily skewed to the left meaning the mean is less than the median. This is negative skewness. For co2 emission, this is skewed to the right with a positive skew.  This mostly may be caused by historically not releasing as much co2 emissions and humanity is only starting to produce higher co2. Democracy score has a near bell curve skew but has a distribution that is more skewed to the left. Meaning, more countries have a negative democracy score than a positive democracy score. Income per person has a significant right skew.  For this, it actually reflects that there are still a significant number of humans who achieve very low amounts of imcome. Life expectancy has a positive skew as well meaning throughout the years there has been development in life_expectancy but there are so many more years when life expectancy was low.<jupyter_code># box plots to view outliers
le_boxplot = gapminder_data.boxplot(column=['life_expectancy'])
le_boxplot.set_ylabel("Age")
le_boxplot;<jupyter_output><empty_output><jupyter_text>We see the mean for life expectancy to be on the lower end of the interquartile range for the box plot.<jupyter_code>cm_boxplot = gapminder_data.boxplot(column=['child_mortality'])
cm_boxplot.set_ylabel("Death of children per 1,000 living births")
cm_boxplot;<jupyter_output><empty_output><jupyter_text>We see the mean for child mortality to be much higher and a large portion of the data to have higher child  mortality rates<jupyter_code>ce_boxplot = gapminder_data.boxplot(column=['co2_emissions'])
ce_boxplot.set_ylabel("Metric tonnes of CO2 per person")
ce_boxplot;<jupyter_output><empty_output><jupyter_text>Overall there are much lower rates of co2 emissions<jupyter_code>ds_boxplot = gapminder_data.boxplot(column=['democracy_score'])
ds_boxplot.set_ylabel("Score")
ds_boxplot;<jupyter_output><empty_output><jupyter_text>There is a significant amount of outliers for democracy score.  This appears to have one significant on the lower end and a good amount on the higher end.  It shows that there are countries with these democracy scores that are creating these outliers.<jupyter_code>ipp_boxplot = gapminder_data.boxplot(column=['income_per_person'])
ipp_boxplot.set_ylabel("income")
ipp_boxplot;<jupyter_output><empty_output><jupyter_text>There is a significant amount of outliers for income per person.  This illustrates that there is indeep large income gaps  Research Question 2: Is there a correlations between the Dependent variable (life expectancy) and the independent variables?<jupyter_code># initial review of correlation
gapminder_data.corr()
heat_map = sns.heatmap(gapminder_data.corr(), annot=True)<jupyter_output><empty_output><jupyter_text>With a view of the data on a heatmap, we can see that life expectancy has a negative correlation with all the other variables including life expectancy. Life Expectancy has a positive correlation with year, income per person, democracy score, and co2 emissions. The strongest correlations with life expectancy in descending order are income per person, co2 emissions, year, and democracy score. This shows the powerful impact these independent variables do have on life expectancy.   Taking a look at the data over time<jupyter_code>le_line = gapminder_data.plot.line(x='year', y ='life_expectancy')
le_line.set_ylabel("age")
le_line.set_title("Life Expectancy")
le_line;<jupyter_output><empty_output><jupyter_text>We see a increase in life expectancy throughout the years with a sharp dip during the world wars.<jupyter_code>cm_line = gapminder_data.plot.line(x='year', y ='child_mortality')
cm_line.set_ylabel("Death of children per 1,000 living births")
cm_line.set_title("Child Mortality")
cm_line;<jupyter_output><empty_output><jupyter_text>Child mortality has decreased throughout the years and had a much sharper decline in the 1900s<jupyter_code>ce_line = gapminder_data.plot.line(x='year', y ='co2_emissions')
ce_line.set_ylabel("Metric tonnes of CO2 per person")
ce_line.set_title("CO2 Emissions")
ce_line;<jupyter_output><empty_output><jupyter_text>Co2 Emissions have increased overtime and we see high development in the 1940s.  This could have been during the World War 2 era where gun power and weaponry were being developed.<jupyter_code>ds_line = gapminder_data.plot.line(x='year', y ='democracy_score')
ds_line.set_ylabel("Score")
ds_line.set_title("Democracy Score")
ds_line;<jupyter_output><empty_output><jupyter_text>We see a sharp decline in democracy score after world war 1 and have remained low past world war 2. In the 1980s this was the time when decocracy started to have an increase again.<jupyter_code>ipp_line = gapminder_data.plot.line(x='year', y ='income_per_person')
ipp_line.set_ylabel("Income")
ipp_line.set_title("Income Per Person")
ipp_line;<jupyter_output><empty_output><jupyter_text>There is a sharp increase in income per person starting in the 1950s and I believe this is associated with the 1950s - 1970s industrial revolution  Research Question 2: Closer review of the correlation<jupyter_code>child_mortality_le = gapminder_data.plot.scatter(x='life_expectancy'
                                                ,y='child_mortality')
child_mortality_le.set_title("Child Mortality & Life Expectancy Correlation");<jupyter_output><empty_output><jupyter_text>Clear negative correlation with child mortality and life expectancy<jupyter_code>co2_emissions_le = gapminder_data.plot.scatter(x='life_expectancy'
                                                ,y='co2_emissions')
co2_emissions_le.set_title("CO2 Emissions & Life Expectancy Correlation");<jupyter_output><empty_output><jupyter_text>Positive correlation with co2 emissions and life expectancy<jupyter_code>democracy_score_le = gapminder_data.plot.scatter(x='life_expectancy'
                                                ,y='democracy_score')
democracy_score_le.set_title("Democracy Score vs Life Expectancy Correlation");<jupyter_output><empty_output><jupyter_text>Slight neutral correlation but still positive correlation with democracy score and life expectancy<jupyter_code>income_per_person_le = gapminder_data.plot.scatter(x='life_expectancy'
                                                ,y='income_per_person')
income_per_person_le.set_title("Income Per Person & Life Expectancy Correlation");<jupyter_output><empty_output><jupyter_text>Positive correlation with income per person and life expectancy.   Limitations> There are metrics of the data that are arbritrary that may be effecting the conculsions and results. > Limitation 1: CO2 Emissions is a prediction is difficult to track for earlier years.  This is a limit in the true CO2 Emissions since tracking of this data is only an estimate. > Limitation 2: With the growth of CO2 Emissions leading to more Life Expectancy, we have a missing element of data of how many CO2 Emissions solutions have caused death rates vs when we didn't have these solutions. That is another limitation on the true outcome of CO2 Emissions. > Limitation 3: Democracy Score is also an estimate and arbitrary and because of this is, it is hard to define if it does have a true impact. Democracy Score and Corruption Preception would have been good metrics to go hand in hand to see if these two arbitrary metrics may have a trend > Limitation 4: The data is taking an average across all available countries, meaning there is a good amount of countries missing.  There could also be a skew in data because the metrics are coming from mostly developed countries.  For a true analysis, there would need to be all countries available in the world to have all the variables.   Conclusions> Overtime life expectancy is definitely improving!  We see the strongest correlation with income per person and life expectancy.  This means enabling more people with opportunities to have income will lead to them living longer. > We also see a strong correlation with co2 emissions which means that more technological developments that have advanced over the years have made human living easier and improved life expectancy.  This doesn't mean we should stick to co2 but then develop with alternative methods. > Democracy score also has a positive correlation but is close to neutral.  Meaning it may not have that much of an impact on life expectancy and is only slight correlated. > Child mortality has a negative correlation meaning that the lower the rate of children dying, the higher life expectancy will be which is expected. > It was great to do this analysis and see how significant moments in history have played an impact in advancing life expectancy.  I predict life expectancy will continue to increase as the data shows continuous increase.   Submitting your Project > Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).> Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.> Once you've done this, you can submit your project by clicking on the "Submit Project" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!<jupyter_code>from subprocess import call
call(['python', '-m', 'nbconvert', 'Investigate_a_Dataset.ipynb'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.056298118049615056, 'fraction_numerical': 0.011174080410607357, 'mean_word_length': 5.6165546515741065, 'pattern_counts': {'":': 1, '<': 117, '<?xml version=': 0, '>': 135, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '55653', 'license': 'MIT', 'n_tokens_mistral': 5381, 'n_tokens_neox': 5016, 'n_words': 2150, 'path': 'Investigate_a_Dataset_Assignment/Investigate_a_Dataset.ipynb', 'repo_name': 'christineton/Udacity_Data_Analysis_Nanodegree', 'size': '18704'}