<jupyter_start><jupyter_text>Purpose In this notebook, we will investigate how Covid-19 affect the happiness of the world. Three datasets will be used:The first two are datasets from the world happiness report from 2019 to 2021 respectively.The last dataset is the worldwide Covid-19 data.  Questions we care about: 1/ What are the happiness scores across countries before Covid?2/ What are the happiness scores across countries after Covid?3/ How many people die due to Covid?4/ How did Covid affect happiness score? To answer these questions, we will take a look at the happiness scores of the countries before and after Covid.  Loading Necessary Modules and Data<jupyter_code>import pandas as pd
import numpy as np
import os
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale
import warnings
warnings.filterwarnings('ignore')
# the supplied CSV data file is the raw_data directory
world_happiness_2019 = pd.read_csv('2019.csv')
world_happiness_2021 = pd.read_csv('2021.csv')
covidtill2021 = pd.read_csv('worldwide covid data.csv')<jupyter_output><empty_output><jupyter_text>Data Wrangling Before doing any analysis, let's take a look at the data and check if there is any null or duplicate.<jupyter_code>world_happiness_2019.head()
world_happiness_2019.describe(include = 'all')<jupyter_output><empty_output><jupyter_text>Glancing from the description of the dataset, it seems there is no null value. Now, let's check 2021 world happiness dataset.<jupyter_code>world_happiness_2021.head()
world_happiness_2021.describe(include = 'all')<jupyter_output><empty_output><jupyter_text>It seems there is no null value in the 2021. However, many columns in the 2021 dataset are not in the 2019 dataset. Some columns are also named differently. In addition, the number of countries of the 2019 dataset is smaller than the 2021 dataset. We need to take a closer look at this. Since we want to directly compare the two datasets, we will drop the columns in the 2021 dataset that are not in the 2019 dataset.<jupyter_code>columnstodrop=['Standard error of ladder score','upperwhisker','lowerwhisker','Ladder score in Dystopia','Explained by: Log GDP per capita'
              ,'Explained by: Social support','Explained by: Healthy life expectancy','Explained by: Generosity','Explained by: Perceptions of corruption'
              ,'Explained by: Perceptions of corruption','Dystopia + residual','Explained by: Freedom to make life choices']
world_happiness_2021.drop(columns=columnstodrop,inplace=True)
world_happiness_2019.drop(columns=['Overall rank'],inplace=True)<jupyter_output><empty_output><jupyter_text>Now, let's rename some of the columns so they are consistent across the datasets.<jupyter_code>world_happiness_2021.rename(columns={"Country name":"Country or region","Ladder score":"Score"},
                            inplace=True)
world_happiness_2021.head()
world_happiness_2019.head()
country_2019=set(world_happiness_2019['Country or region'].to_list())
country_2021=set(world_happiness_2021['Country or region'].to_list())<jupyter_output><empty_output><jupyter_text>Now, let's check which countries are in the 2021 dataset and not the 2019 dataset.<jupyter_code>country_2021.difference(country_2019)<jupyter_output><empty_output><jupyter_text>Now, let's check which countries are in the 2019 dataset and not the 2021 dataset.<jupyter_code>country_2019.difference(country_2021)<jupyter_output><empty_output><jupyter_text>It seems some of the names of some countries are inconsistent across the datasets. Some countries in one dataset are not listed in the other. We will rename the countries with inconsistency and drop the rest.<jupyter_code>world_happiness_2021.loc[world_happiness_2021['Country or region']=='Hong Kong S.A.R. of China','Country or region']='Hong Kong'
world_happiness_2021.loc[world_happiness_2021['Country or region']=='Taiwan Province of China','Country or region']='Taiwan'
world_happiness_2021.loc[world_happiness_2021['Country or region']=='North Cyprus','Country or region']='Northern Cyprus'
world_happiness_2021.drop(world_happiness_2021.index[world_happiness_2021['Country or region']=='Maldives'].tolist(),inplace=True)
country_2021=set(world_happiness_2021['Country or region'].to_list())
country_2019=set(world_happiness_2019['Country or region'].to_list())
countrytodrop=list(country_2019.difference(country_2021))
for i in countrytodrop:
    world_happiness_2019.drop(world_happiness_2019.index[world_happiness_2019['Country or region']==i],inplace=True)
country_2019=set(world_happiness_2019['Country or region'].to_list())
country_2021=set(world_happiness_2021['Country or region'].to_list())
country_2021.difference(country_2019)
country_2019.difference(country_2021)<jupyter_output><empty_output><jupyter_text>Let's also add a column in 2019 on the region of the country using the 2021 dataset.<jupyter_code>world_happiness_2019=world_happiness_2019.merge(world_happiness_2021[['Country or region','Regional indicator']], on='Country or region', how='left')<jupyter_output><empty_output><jupyter_text>Next, let's check the dataset that contains information about Covid.<jupyter_code>covidtill2021.head()
covidtill2021.describe(include = 'all')
countrycovid_2021=set(covidtill2021['Country'].to_list())
countrycovid_2021.difference(country_2021)
country_2021.difference(countrycovid_2021)<jupyter_output><empty_output><jupyter_text>Similar to the world_happiness_2021 and the world_happiness_2019 datasets, the covidtill2021 dataset has many names that are not consistent with the world_happiness_2021 dataset. We will rename what we can and drop the rest.<jupyter_code>covidtill2021.loc[covidtill2021['Country']=='Congo','Country']='Congo (Brazzaville)'
covidtill2021.loc[covidtill2021['Country']=='Czechia','Country']='Czech Republic'
covidtill2021.loc[covidtill2021['Country']=='Palestine','Country']='Palestinian Territories'
covidtill2021.loc[covidtill2021['Country']=='S. Korea','Country']='South Korea'
covidtill2021.loc[covidtill2021['Country']=='UAE','Country']='United Arab Emirates'
covidtill2021.loc[covidtill2021['Country']=='UK','Country']='United Kingdom'
covidtill2021.loc[covidtill2021['Country']=='USA','Country']='United States'
countrycovid_2021=set(covidtill2021['Country'].to_list())
for i in list(country_2021.difference(countrycovid_2021)):
    world_happiness_2021.drop(world_happiness_2021.index[world_happiness_2021['Country or region']==i],inplace=True)
for i in list(country_2021.difference(countrycovid_2021)):
    world_happiness_2019.drop(world_happiness_2019.index[world_happiness_2019['Country or region']==i],inplace=True)
country_2021=set(world_happiness_2021['Country or region'].to_list())
country_2019=set(world_happiness_2019['Country or region'].to_list())
for i in list(countrycovid_2021.difference(country_2021)):
    covidtill2021.drop(covidtill2021.index[covidtill2021['Country']==i],inplace=True)
countrycovid_2021=set(covidtill2021['Country'].to_list())
countrycovid_2021.difference(country_2021)
covidtill2021.head()<jupyter_output><empty_output><jupyter_text>Let's also add a column to indicate the region of the country in the Covid dataset.<jupyter_code>covidtill2021.rename(columns={"Country":"Country or region"},
                            inplace=True)
covidtill2021=covidtill2021.merge(world_happiness_2021[['Country or region','Regional indicator']], on='Country or region', how='left')
covidtill2021.head()<jupyter_output><empty_output><jupyter_text>EDA So with this, let's answer the first question we have: What are the happiness scores across countries before Covid?<jupyter_code>world_happiness_2019[['Country or region','Score']].sort_values(by=['Score'],ascending=False).head()
world_happiness_2019[['Country or region','Score']].sort_values(by=['Score'],ascending=True).head()<jupyter_output><empty_output><jupyter_text>It seems that Finland is the happiest country in the world while Afghanistan is the least. Let's see how the happiness score distribute.<jupyter_code>plt.hist(world_happiness_2019['Score'],alpha=0.5, label='2019')
plt.show()<jupyter_output><empty_output><jupyter_text>Next, let's see how this score spread across different regions.<jupyter_code>plt.rcParams['figure.figsize'] = (15, 12)
sns.violinplot(world_happiness_2019['Score'], world_happiness_2021['Regional indicator'])
plt.show()<jupyter_output><empty_output><jupyter_text>It seems the most happy regions are from Western Europe and North America. Now, let's answer the second question we have: What are the happiness scores across countries after Covid?<jupyter_code>world_happiness_2021[['Country or region','Score']].sort_values(by=['Score'],ascending=False).head()
world_happiness_2021[['Country or region','Score']].sort_values(by=['Score'],ascending=True).head()<jupyter_output><empty_output><jupyter_text>So even after Covid, Finland is the happiest country in the world while Afghanistan is the least. Let's see how the happiness score distribute.<jupyter_code>plt.hist(world_happiness_2021['Score'],alpha=0.5, label='2021')
plt.show()
plt.rcParams['figure.figsize'] = (15, 12)
sns.violinplot(world_happiness_2021['Score'], world_happiness_2021['Regional indicator'])
plt.show()<jupyter_output><empty_output><jupyter_text>Now, let's compare the distribution of the happiness score directly before and after Covid.<jupyter_code>plt.hist(world_happiness_2021['Score'],alpha=0.5, label='2021')
plt.hist(world_happiness_2019['Score'],alpha=0.5, label='2019')
plt.legend(loc='upper right')
plt.show()<jupyter_output><empty_output><jupyter_text>We see that Covid did not seem to change the happiness score of many countries much. But it does widen the distribution, making some happy countries more happy and some distressed countries more distressed.<jupyter_code>g=sns.jointplot(y=world_happiness_2021['Score'], x=world_happiness_2019['Score'])
x0, x1 = g.ax_joint.get_xlim()
y0, y1 = g.ax_joint.get_ylim()
lims = [max(x0, y0), min(x1, y1)]
g.ax_joint.plot(lims, lims, '-r')
g.set_axis_labels('Score 2019', 'Score 2021')
plt.show()<jupyter_output><empty_output><jupyter_text>Since the dots follow the diagoline quite closely, it seems Covid does not seem to affect the happiness score of the world much. Next, let's directly compare how the score shift.<jupyter_code>worldhappinesdiff=world_happiness_2019[['Country or region','Regional indicator','Score']]
worldhappinesdiff=worldhappinesdiff.merge(world_happiness_2021[['Country or region','Score']], on='Country or region', how='left')
worldhappinesdiff.head()
worldhappinesdiff.rename(columns={"Score_x":"Score_2019","Score_y":"Score_2021"},
                            inplace=True)
worldhappinesdiff['score diff']=worldhappinesdiff['Score_2021']-worldhappinesdiff['Score_2019']
plt.hist(worldhappinesdiff['score diff'],alpha=0.5)
plt.show()
worldhappinesdiff['score diff'].mean()<jupyter_output><empty_output><jupyter_text>The happiness score mean actually increases by a small amount. It seems that by 2021 most countries happiness scores have stabilized back to pre-Covid era. Let's take a look at how this score change distribute in different regions.<jupyter_code>sns.violinplot(worldhappinesdiff['score diff'], worldhappinesdiff['Regional indicator'])
plt.axvline(0, 0,2)
plt.show()<jupyter_output><empty_output><jupyter_text>It seems that most country in North America, Latin America, South Asia, Middle East have their happiness scores down. While the other regions have their score up.  Now, let's use the Covid dataset to answer the question: how many people die due to Covid?<jupyter_code>covidtill2021[['Country or region','Regional indicator','Total Cases/1M population']].sort_values(by=['Total Cases/1M population'],ascending=False).head()
covidtill2021[['Country or region','Regional indicator','Total Cases/1M population']].sort_values(by=['Total Cases/1M population'],ascending=True).head()<jupyter_output><empty_output><jupyter_text>Hmm, in term of cases per 1M population, Montenegro has the biggest cases while China has the smallest.<jupyter_code>covidtill2021[['Country or region','Regional indicator','Deaths/1M population']].sort_values(by=['Deaths/1M population'],ascending=False).head()
covidtill2021[['Country or region','Regional indicator','Deaths/1M population']].sort_values(by=['Deaths/1M population'],ascending=True).head()<jupyter_output><empty_output><jupyter_text>In term of deaths per 1M population, Peru is the highest while China is the least again.<jupyter_code>covidtill2021.head()
plt.hist(covidtill2021['Total Cases/1M population'],alpha=0.5)
plt.show()
sns.violinplot(covidtill2021['Total Cases/1M population'], covidtill2021['Regional indicator'])
plt.show()
sns.violinplot(covidtill2021['Deaths/1M population'], covidtill2021['Regional indicator'])
plt.show()<jupyter_output><empty_output><jupyter_text>According to these graphs, the regions that are most devastated by Covid are Central, Eastern Europe and Latin America. It's rather strange that even though Covid originated from Asia, Asia's death tolls are rather small compared to other regions. Perhaps a better indicator of Covid is the number of cases per test.<jupyter_code>covidtill2021['Cases per Test']=covidtill2021['Total Cases']/covidtill2021['Total Tests']
sns.violinplot(covidtill2021['Cases per Test'], covidtill2021['Regional indicator'])
plt.show()<jupyter_output><empty_output><jupyter_text>Using the number of cases per test, we see that most the regions fall in line much better. The values of Asia regions are now comparable to regions in America. Now, let's check how Covid affect the happiness scores by using a correlation heat map.<jupyter_code>covidtill2021=covidtill2021.merge(worldhappinesdiff[['Country or region','score diff']], on='Country or region', how='left')
corrmat = covidtill2021.corr()
f, ax = plt.subplots(figsize=(16, 12))
hm = sns.heatmap(round(corrmat,2), annot=True, ax=ax, cmap="coolwarm",fmt='.2f',
                 linewidths=.05)
f.subplots_adjust(top=0.93)
plt.savefig("Feature_correlation.png", bbox_inches="tight")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11156732266742178, 'fraction_numerical': 0.052314880362452215, 'mean_word_length': 8.374253483742535, 'pattern_counts': {'":': 5, '<': 105, '<?xml version=': 0, '>': 105, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '283823', 'license': 'MIT', 'n_tokens_mistral': 5012, 'n_tokens_neox': 4379, 'n_words': 1126, 'path': '.ipynb_checkpoints/Covid-19 Influence on World Happiness-checkpoint.ipynb', 'repo_name': 'mnguyen494/World-Happiness', 'size': '14126'}