<jupyter_start><jupyter_text>Analyzing the World Happiness Data Grouping together subsets of data via split-apply-combine In this exercise, we will apply the split-apply-combine methodology to take advantage of substructure within the WHR panel data.As before, we'll repeat here some of the code developed in previous exercises so that we can continue to work with data in this exercise. Execute the next few code cells to load and reconfigure the data. In addition to the full dataset, we will also want to examine the data just from 2015-2017, which we will store in a dataframe named `df1517`.<jupyter_code>import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
pd.options.display.float_format = '{:.2f}'.format
dfraw = pd.read_excel('WHR2018Chapter2OnlineData.xls', sheet_name='Table2.1')
cols_to_include = ['country', 'year', 'Life Ladder', 
                   'Positive affect','Negative affect',
                   'Log GDP per capita', 'Social support',
                   'Healthy life expectancy at birth', 
                   'Freedom to make life choices', 
                   'Generosity', 'Perceptions of corruption']
renaming = {'Life Ladder': 'Happiness', 
            'Log GDP per capita': 'LogGDP', 
            'Social support': 'Support', 
            'Healthy life expectancy at birth': 'Life', 
            'Freedom to make life choices': 'Freedom', 
            'Perceptions of corruption': 'Corruption', 
            'Positive affect': 'Positive', 
            'Negative affect': 'Negative'}
df = dfraw[cols_to_include].rename(renaming, axis=1)
df1517 = df[df.year.isin(range(2015,2018))]<jupyter_output><empty_output><jupyter_text>Step 1First examine the head of the ```df``` dataframe to remind yourself of its structure.  Each row contains information about a single country in a single year.  We will want to group some of these data elements together.<jupyter_code>df<jupyter_output><empty_output><jupyter_text>Step 2In the previous video, we examined how to group data together for each country and compute an aggregrate property such as the mean over that group.  In the code cell below, enter and evaluate an expression to compute the mean value of the WHR indicators for each country, averaged over all years, and assign the result to the variable ```dfmean```.  To do this, you will want to use the ```groupby``` method on the ```df``` dataframe.  Examine ```dfmean``` to verify that it has the correct structure, which should consist of one row per country (unlike the raw data we examined above).<jupyter_code>df.groupby('country').mean()<jupyter_output><empty_output><jupyter_text>Step 3Now let's look at variation over time.  The code cell below uses the ```groupby``` method. The code:* computes the mean value of all WHR indicators per year (over all countries), and assigns the result to the variable ```mean_vals_by_year```* computes the minimum values of all WHR indicators per year (over all countries), and assigns the result to the variable ```min_vals_by_year```* computes the maximum values of all WHR indicators per year (over all countries), and assigns the result to the variable ```max_vals_by_year```<jupyter_code>mean_vals_by_year = df.groupby('year').mean()
min_vals_by_year = df.groupby('year').min()
max_vals_by_year = df.groupby('year').max()<jupyter_output><empty_output><jupyter_text>Step 4The code cell below contains code to plot the min values of Happiness by year, using the dataframes computed in the cell above. Add code to plot the mean and max values as well.  Plot all three curves in the same figure. The following are good references: the [plot function](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.html) for pandas dataframes. [This tutorial](https://pandas.pydata.org/docs/getting_started/intro_tutorials/04_plotting.html) and [this tutorial](https://pandas.pydata.org/pandas-docs/stable/user_guide/visualization.html). Note: a self-check will not accompany this exercise.Your plot should look like this (although the colors may be different depending on the order in which you plot each curve):<jupyter_code>min_vals_by_year['Happiness'].plot()
max_vals_by_year['Happiness'].plot()
mean_vals_by_year['Happiness'].plot()<jupyter_output><empty_output><jupyter_text>Step 5The World Happiness Report introduces a hypothetical country named Dystopia, "so named because it has values equal to the world’s lowest national averages for 2015-2017 for each of the six key variables used in Table 2.1", where the six key variables are 'LogGDP', 'Support', 'Life', 'Freedom', 'Generosity', and 'Corruption'.Below, you will use the ```groupby``` method to compute the parameters associated with Dystopia, that is, the values of 'LogGDP', 'Support', 'Life', 'Freedom', 'Generosity', and 'Corruption' associated with this hypothetical country, which is at the bottom of the heap for all indicators.You'll need to do this in a few steps, because of the nature of the data.  Whereas 5 of the 6 key variables are generally positive attributes that one would want more of, the last &mdash; Perceptions of Corruption &mdash; is a negative attribute.  So while Dystopia is associated with the world's *lowest* national averages for the first 5 of the key variables, it is associated with the world's *highest* national average for the last variable.In the code cell below:* Compute the mean values of all WHR indicators by country over the 2015-2017 time frame, and then compute the lowest national averages over all countries.  Assign this result to the variable ```min_avg_vals```.* Compute the mean values of all WHR indicators by country over the 2015-2017 time frame, and then compute the highest national averages over all countries.  Assign this result to the variable ```max_avg_vals```.* Assign to the variable ```explanatory_vars``` a list of the six key variable names.* You'll want to create a variable named ```Dystopia``` that contains the data in ```min_avg_vals``` for the first 5 of the explanatory variables, and the data in ```max_avg_vals``` for the last variable.  There are a couple different ways to do this, but we'll just carry out one of them here.* Make a copy of the ```min_avg_vals``` (using the ```copy``` method on that Series), extract out just the columns associated with the explanatory variables from that copy, and assign the result to the variable ```Dystopia```.  The Series ```Dystopia``` should now have all the same values as the data in ```min_avg_vals```, but since we made a copy, we can now modify one of the entries in ```Dystopia``` without changing the data in ```min_avg_vals``` as well.* Now overwrite the ```Corruption``` entry in ```Dystopia``` by assigning it to the corresponding value from ```max_avg_vals```.* Print out the value of Dystopia.<jupyter_code># YOUR CODE HERE
min_avg_vals = df1517.groupby('country').min()
max_avg_vals = df1517.groupby('country').max()
explanatory_vars = df1517.groupby('country')
Dystopia = min_avg_vals.copy('country')
Corruption = max_avg_vals.copy('country')
print(Dystopia)<jupyter_output>year  Happiness  Positive  Negative  LogGDP  Support  Life  \
country                                                                   
Afghanistan  2015       2.66      0.50      0.34    7.46     0.49 51.69   
Albania      2015       4.51      0.67      0.32    9.30     0.64 68.69   
Algeria      2016       5.25      0.64      0.29    9.54     0.75 65.51   
Argentina    2015       6.04      0.81      0.29    9.82     0.88 67.26   
Armenia      2015       4.29      0.59      0.44    9.01     0.70 64.80   
...           ...        ...       ...       ...     ...      ...   ...   
Venezuela    2015       4.04      0.69      0.22    9.53     0.90 64.60   
Vietnam      2015       5.06      0.54      0.22    8.64     0.85 65.91   
Yemen        2015       2.98      0.46      0.23    7.75     0.67 54.61   
Zambia       2015       3.93      0.68      0.37    8.20     0.69 52.73   
Zimbabwe     2015       3.64      0.72      0.18    7.54     0.74 50.93   

             Freedom  Ge[...]<jupyter_text>Self-CheckRun the cell below to test the correctness of your code above before submitting for grading.<jupyter_code># Run this self-test cell to check your code; do not add code or delete code in this cell
from jn import testDystopia

try:
    print(testDystopia(df1517, Dystopia, min_avg_vals, max_avg_vals, explanatory_vars))
except Exception as e:
    print("Error!\n" + str(e))<jupyter_output>Incorrect!
Check your values for min_avg_vals and max_avg_vals.
Check your values for explanatory_vars array.
Dystopia does not contain the correct values.
{'fraction_non_alphanumeric': 0.08711203415253259, 'fraction_numerical': 0.03646013614860967, 'mean_word_length': 3.505197505197505, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '234512', 'license': 'Apache-2.0', 'n_tokens_mistral': 2698, 'n_tokens_neox': 2414, 'n_words': 1051, 'path': 'GroupingKeys.ipynb', 'repo_name': 'pujaroy280/AI-Foundations', 'size': '8667'}