<jupyter_start><jupyter_text>Data Analysis Tools Assignment: Generating a Correlation CoefficientFollowing is the Python program I wrote to fulfill the third assignment of the [Data Analysis Tools online course](https://www.coursera.org/learn/data-analysis-tools/home/welcome).I decided to use [Jupyter Notebook](http://nbviewer.jupyter.org/github/ipython/ipython/blob/3.x/examples/Notebook/Index.ipynb) as it is a pretty way to write code and present results. Research questionUsing the [Gapminder database](http://www.gapminder.org/), I would like to see if an increasing Internet usage results in an increasing suicide rate. A study shows that other factors like unemployment could have a great impact.So for this assignment, the three following variables will be analyzed:- Internet Usage Rate (per 100 people)- Suicide Rate (per 100 000 people)- Unemployment Rate (% of the population of age 15+) Data managementFor the question I'm interested in, the countries for which data are missing will be discarded. As missing data in Gapminder database are replace directly by `NaN` no special data treatment is needed.<jupyter_code># Magic command to insert the graph directly in the notebook
%matplotlib inline
# Load a useful Python libraries for handling data
import pandas as pd
import numpy as np
import statsmodels.formula.api as smf
import scipy.stats as stats
import seaborn as sns
import matplotlib.pyplot as plt
from IPython.display import Markdown, display
# Read the data
data_filename = r'gapminder.csv'
data = pd.read_csv(data_filename, low_memory=False)
data = data.set_index('country')<jupyter_output><empty_output><jupyter_text>General information on the Gapminder data<jupyter_code>display(Markdown("Number of countries: {}".format(len(data))))
display(Markdown("Number of variables: {}".format(len(data.columns))))
# Convert interesting variables in numeric format
for variable in ('internetuserate', 'suicideper100th', 'employrate'):
    data[variable] = pd.to_numeric(data[variable], errors='coerce')<jupyter_output><empty_output><jupyter_text>But the unemployment rate is not provided directly. In the database, the employment rate (% of the popluation) is available. So the unemployement rate will be computed as `100 - employment rate`:<jupyter_code>data['unemployrate'] = 100. - data['employrate']<jupyter_output><empty_output><jupyter_text>The first records of the data restricted to the three analyzed variables are:<jupyter_code>subdata = data[['internetuserate', 'suicideper100th', 'unemployrate']]
subdata.tail(10)<jupyter_output><empty_output><jupyter_text>Data analysisThe distribution of the three variables have been analyzed [previously](Visualizing_Data.ipynb).  Variance analysisAs all variables are quantitative, the Pearson correlation test is the one to apply.Let's first focus on the primary research question;- The explanatory variable is the internet use rate (quantitative variable)- The response variable is the suicide per 100,000 people (quantitative variable)From the scatter plot, a slope slightly positive is seen. But will the Pearson test confirm this is significant?<jupyter_code>sns.regplot(x='internetuserate', y='suicideper100th', data=subdata)
plt.xlabel('Internet use rate (%)')
plt.ylabel('Suicide per 100 000 people (-)')
_ = plt.title('Scatterplot for the association between the Internet use rate and suicide per 100,000 people')
data_clean = subdata.dropna()
correlation, pvalue = stats.pearsonr(data_clean['internetuserate'], data_clean['suicideper100th'])

display(Markdown("The correlation coefficient is {:.3g} and the associated p-value is {:.3g}.".format(correlation, pvalue)))<jupyter_output><empty_output><jupyter_text>The correlation coefficient is 0.0735 confirming the small positive correlation. But the Pearson test tells us that **the null hypothesis cannot be rejected** as the p-value is 0.351 >> 0.05.This confirms the conclusion found when grouping the internet use rate in quartile and applying [ANOVA test](Analysis_Variance.ipynb). If we look now at the relationship between unemployment and suicide, it seems that there is no relationship looking at the scatterplot below.<jupyter_code>sns.regplot(x='unemployrate', y='suicideper100th', data=subdata)
plt.xlabel('Unemployment rate (%)')
plt.ylabel('Suicide per 100 000 people (-)')
_ = plt.title('Scatterplot for the association between the unemployment rate and suicide per 100,000 people')<jupyter_output><empty_output><jupyter_text>Does the Pearson test confirms that conclusion?<jupyter_code>correlation, pvalue = stats.pearsonr(data_clean['unemployrate'], data_clean['suicideper100th'])

display(Markdown("The correlation coefficient is {:.3g} and the associated p-value is {:.3g}.".format(correlation, pvalue)))<jupyter_output><empty_output><jupyter_text>The correlation coefficient is negative but really small and the p-value is large. So we can safetly conclude that there is no relationship between the unemployment rate and the suicide per 100,000 people.  Another test caseIn order to look at the coefficient of determination, an another relationship that is significant will be analyzed below: Is the residential electricity consumption (response variable) related to the income per person (explanatory variable)?<jupyter_code>subdata2 = (data[['incomeperperson', 'relectricperperson']]
                .assign(income=lambda x: pd.to_numeric(data['incomeperperson'], errors='coerce'),
                        electricity=lambda x: pd.to_numeric(data['relectricperperson'], errors='coerce'))
                .dropna())

sns.regplot(x='income', y='electricity', data=subdata2)
plt.xlabel('Income per person (2000 US$)')
plt.ylabel('Residential electricity consumption (kWh)')
_ = plt.title('Scatterplot for the association between the income and the residential electricity consumption')
correlation, pvalue = stats.pearsonr(subdata2['income'], subdata2['electricity'])

display(Markdown("The correlation coefficient is {:.3g} and the associated p-value is {:.3g}.".format(correlation, pvalue)))
display(Markdown("And the coefficient of determination is {:.3g}.".format(correlation**2)))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09315157416423239, 'fraction_numerical': 0.016553067185978577, 'mean_word_length': 6.407451923076923, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '494480', 'license': 'MIT', 'n_tokens_mistral': 1803, 'n_tokens_neox': 1654, 'n_words': 624, 'path': 'PearsonCorrelation.ipynb', 'repo_name': 'fcollonval/coursera_data_visualization', 'size': '6162'}