<jupyter_start><jupyter_text>Author <NAME>  AbstractIn this project I will focus on a dataset "Health nutrition and population statistics". The dataset is available in "data" folder, but is also available online at Kaggle( for the exact link please refer to the Resources section).  The structure of the project is as it follows: 1. Abstract 2. Correlation analysis 3. EDA  4. Hypothesis testing 5. Conclusion 6. Further development 7. Resources I will start with reading and cleaning the dataset. Then I will check on the correLation coefficient and perform some EDA. Eventually, I will perform a hypothesis testing to define if there is strong connection between the data from 20th century and 21st century. Please note that for faster performing I will clean the dataset and reduce it. As usual, I will check the shape, dtypes, and check for some unique values of the data.<jupyter_code>health_data = pd.read_csv('data/health_nutrition_and_population.csv')
health_data.shape
health_data.head()
health_data.columns.values<jupyter_output><empty_output><jupyter_text>I will check on the unique values of the countries, and the indicator name columns. I will group the dataset<jupyter_code>countries = health_data["Country Name"].unique()
len(countries)
indicator_name = health_data["Indicator Name"].unique()
indicator_name
len(indicator_name)<jupyter_output><empty_output><jupyter_text>I notice something interesting here - there are 258 unique countries, each of which present in the dataset 345 times. The Indicator names column is exactly the opposite - there are 345 unique values, each of which presents in the dataset 258 times. Once I have reviewed the dataset, I will call the describe function also.<jupyter_code>health_data.info()
display(health_data.describe(include="all").T)<jupyter_output><empty_output><jupyter_text>Dataset is quite long ( it contains information for 55 years - 1960 to 2015 including). Thus, I will choose to focus to compare the values from less years - with step of 11 years, and try to find some trend. Lets clean the dataset accordingly by dropping the not needed informaiton. I will show the column values from the cleaned data, as well as the types of the columns. I could also save the new file as .csv, but I don't need it so I will skip this step.<jupyter_code>health_data= health_data.drop(['1961', '1962', '1963', '1964','1965', '1966', '1967',
       '1968', '1969','1970', '1972', '1973', '1974', '1975',
       '1976', '1977', '1978', '1979', '1980', '1981', '1983',
       '1984', '1985', '1986', '1987', '1988', '1989','1990','1991', '1991',
       '1992',  '1994', '1995', '1996', '1997', '1998', '1999','2000',
       '2001', '2002', '2003',  '2005', '2006', '2007',
       '2008', '2009', '2010','2011', '2012', '2013', '2014', 'Unnamed: 60'], axis=1)
health_data.columns.values
health_data.shape
health_data.dtypes<jupyter_output><empty_output><jupyter_text>We can see that the dataset is significantly reduced now. This gives me the opportunity to focus on specific part of it, and to have better performance.I didn't clean the dataset from NaN values on purpose. This leads to loss of information, which leads to different results in the end. Thus I will clean the data at later stage ( plus this is quite good way to have the oppotunity to change, instead drop the NaN values if needed.)  Correlation coefficient Data is already cleaned. Lets theck on the correlation coefficient now.<jupyter_code>plt.figure(figsize=(20,10))
sns.heatmap(health_data.corr(), annot = True)
plt.show()<jupyter_output><empty_output><jupyter_text>We see that the correlation is quite strong at all cells  ( the lowest one is 0.96). This actually leads me to the conclusion, that this correlation matrix does not give me actually that much of information. I will try to plot some scatterplots to check the relation between few years.<jupyter_code>plt.scatter(health_data['1993'], health_data['2004'])
plt.xlabel('year 1993')
plt.ylabel('year 2004')
plt.title('Correlation for years 1993 and 2004')
plt.show()
plt.scatter(health_data['1960'], health_data['2015'])
plt.xlabel('year 1960')
plt.ylabel('year 2015')
plt.title('Correlation for years 1960 and 2015')
plt.show()
plt.scatter(health_data['2004'], health_data['2015'])
plt.xlabel('year 2004')
plt.ylabel('year 2015')
plt.title('Correlation for years 2004 and 2015')
plt.show()<jupyter_output><empty_output><jupyter_text>As expected, all 3 graphics above visualize the same result - a strong trend exists. Lets make a comparison  of the data from one year and countries.<jupyter_code>plt.figure(figsize=[22,12])
plt.scatter(health_data['Country Name'], health_data['2015'])
plt.xlabel('Country name')
plt.ylabel('year 2015')
plt.title('Correlation for countries and year 2015')
plt.show()
plt.figure(figsize=[22,12])
plt.scatter(health_data['Indicator Name'], health_data['2015'])
plt.xlabel('Indicator Name')
plt.ylabel('year 2015')
plt.title('Correlation for indicator name and year 2015')
plt.show()<jupyter_output><empty_output><jupyter_text>I conclude that the correlation between the latest year, and the countries/indicator data is not that well defined and strong. We have some outliers,which are well visible on both graphics. For better analysis  here I need to define a smaller sample size - focus on particular countries , check the trend and analyze on the root reasons.The countries names and indicator names are not visualized well at all on the graphic. This can be improved in the further development -  by grouping the countries to continents , for example.  EDA analysis<jupyter_code>plt.hist(health_data['1960'].dropna())
plt.title('Year 1960')
plt.xlabel('values for year 1960')
plt.ylabel('Count')
plt.show()
health_data['1960'].skew()
plt.hist(health_data['2015'].dropna())
plt.title('Year 2015')
plt.xlabel('values for year 2015')
plt.ylabel('Count')
plt.show()
health_data['2015'].skew()<jupyter_output><empty_output><jupyter_text>The distribution is definetely not normal. We have very big assymetry . Skewness number is also quite big. BAsed ont the visualizations above, we have very strong positive skewnness.   Now, I will divide the data into 2 groups - one, containing the years from 20th century, and a second one, containing the years from 21st century. I will need them as in the following section I will use the groups to perform a test.<jupyter_code>group_21century = health_data.drop(['1960','1971','1982','1993'],axis=1)
group_20century  = health_data.drop(['2004','2015'], axis=1)
group_20century
group_21century<jupyter_output><empty_output><jupyter_text>Now I will clean a little bit the two groups and drop the NA values. I do not need to replace them.<jupyter_code>group_20century = group_20century.dropna()
group_20century.shape
group_21century = group_21century.dropna()
group_21century.shape<jupyter_output><empty_output><jupyter_text>Hypothesis testingIn this section I will perform a Ztest. The two hypothesis are as it follows:      H0: There is no significant difference between two groups.     H1: There is significant difference between the two groups.    I will directly use the two groups created in the previous section, and take the "border " years - 1960 and 2015.<jupyter_code>z, p = weightstats.ztest(group_20century['1960'].astype(float).dropna(),group_21century['2015'].astype(float))
z, p<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09985076651743319, 'fraction_numerical': 0.06227106227106227, 'mean_word_length': 5.744739249771272, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 46, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '518258', 'license': 'MIT', 'n_tokens_mistral': 2482, 'n_tokens_neox': 2059, 'n_words': 864, 'path': 'Health nutrition and population statistics.ipynb', 'repo_name': 'YanaNeykova/Food-data-analysis', 'size': '7377'}