<jupyter_start><jupyter_text>Altair heatmap and long form data transformation Altair is an information visualization lirary that is relatively easy to understand with clean and straightforward code. The caveat is that the original data needed to be manipulated to appropriate form to be suitable with Altair. In this tutorial, I demonstrate the transformation from a survey data to a long-form dataframe, then use this dataframe to create a complete heatmap with considerations of aesthetic features.  Install and Setup Altair If you have never used Altair before, chances are it is not already installed in your working environment. To install Altair, you can either excecute the following command in pip:```pip install altair``` Or using conda:```conda install -c conda-forge altair``` I am not going to use one of the exisiting datasets in the example [vega_datasets](https://github.com/altair-viz/vega_datasets) in this guide, but if you do, you can install Altair along with these datasets using the following command in pip:```pip install altair vega_datasets``` Or if you are using the conda manager, you do:```conda install -c conda-forge altair vega_datasets``` Now, to make sure your visualizations are rendered correctly in Jupyter Notebook, you want to run the following code:```alt.renderers.enable('default')``` To improve performance when working with large datasets, use the ```json``` data transformer by running the following code:```alt.data_transformers.enable('json')```  Demonstration Now that you already have Altair installed, let's begin creating some awesome visualizations.  I. Import Altair Library and Dataset<jupyter_code># import Altair
import altair as alt

alt.renderers.enable('default')

alt.data_transformers.enable('json')<jupyter_output><empty_output><jupyter_text>I will be using the [Food World Cup](https://github.com/fivethirtyeight/data/tree/master/food-world-cup) dataset by FiveThirtyEight. This is the dataset assocciated with two of their articles : [The FiveThirtyEight International Food Association’s 2014 World Cup](https://fivethirtyeight.com/features/the-fivethirtyeight-international-food-associations-2014-world-cup/) and [What is Americans’ Favorite Global Cuisine?](https://fivethirtyeight.com/features/what-is-americans-favorite-global-cuisine/)<jupyter_code># import Pandas for data manipulation

import pandas as pd
# read the original dataset
df = pd.read_csv('food-world-cup-data.csv')
# take a look at the head of the dataset
df.head()<jupyter_output><empty_output><jupyter_text>II. Manipulate Data to Long Form for Visualization<jupyter_code># remove long question in column names
df.columns = df.columns.str.replace('Please rate how much you like the traditional cuisine of', '')
# remove special characters in column names
df.columns = df.columns.str.replace('[:,.,�]', '')
# remove white spaces in column names
df.columns = df.columns.str.strip()
# rename some of the columns for accessibility
df = df.rename(columns={'RespondentID': 'ID',
                        'Generally speaking how would you rate your level of knowledgeof cuisines from different parts of the world?': 'cuisine_knowledge',
                        'How much if at all are you interested in cuisines from different parts of the world?': 'how_interested',
                        'Location (Census Region)': 'location',
                        'the Netherlands': 'The Netherlands'})
# see the columns that we got
df.columns<jupyter_output><empty_output><jupyter_text>1. First attempt Let's say I want to create a visualization that shows how popular a cuisine is throughout different regions in the U.S. I can do so by creating a heatmap with country names on one axis and locations on the other. Since we have many more country names than we have locations, for legibility, we will want country names on the y-axis and location on the x-axis.<jupyter_code># first I specify all the country names we got
countries = ['Algeria', 'Argentina',
       'Australia', 'Belgium', 'Bosnia and Herzegovina', 'Brazil', 'Cameroon',
       'Chile', 'Colombia', 'Costa Rica', 'Croatia', 'Ecuador', 'England',
       'France', 'Germany', 'Ghana', 'Greece', 'Honduras', 'Iran', 'Italy',
       'Ivory Coast', 'Japan', 'Mexico', 'The Netherlands', 'Nigeria',
       'Portugal', 'Russia', 'South Korea', 'Spain', 'Switzerland',
       'United States', 'Uruguay', 'China', 'India', 'Thailand', 'Turkey',
       'Cuba', 'Ethiopia', 'Vietnam', 'Ireland']
# then I specify all the locations we got
locations = ['East North Central', 'East South Central', 'Middle Atlantic',
       'Mountain', 'New England', 'Pacific', 'South Atlantic',
       'West North Central', 'West South Central']
# create a new dataframe displaying the median score of each cuisine by locations
heatmap_df = df.groupby(['location'])[countries].median().reset_index()
heatmap_df<jupyter_output><empty_output><jupyter_text>My intitial approach was to produce a heatmap that shows how popular a cuisine is by geographical locations. However, after producing the above ```heatmap_df``` DataFrame, I came across a small problem. I realized that the potential graph would contain very little useful information and would not be effective of what I am trying to show because the number of respondents differ among different cuisine. For example, I assume that more people are familar with Italian cuisine versus Algerian cuisine in most locations. Therefore, the median result of each cuisine does not represent how popular the cuisine is. I decided to go with a different approach by showing how popular a cuisine is across **all** locations and how often the repondents rate them on the scale 1 to 5.  2. Second attempt I proceed with this approach by first calculate the total number of repsondents who rated a particular cuisine among all who surveyed. These numbers are expected to be very different because some cuisine (e.g. Italian cuisine) are much more popular than others.<jupyter_code>rate_total = []  # create an empty list contains total number of repondents for each country

for country in countries:    # for each country in the country list specifed above
    count = df.dropna(subset=[country])      # drop NaN from each of the country columns
    total = len(count)                       # count the number of entries excluding NaN responses
    rate_total.append(total)                 # append these number to the list
rate_tuples = list(zip(countries, rate_total))  # create tuples by zipping country names and its corresponding value
rate_total_df = pd.DataFrame(rate_tuples, columns = ['Country', 'Total'])  # make these tuples a dataframe
rate_total_df  # here is our dataframe<jupyter_output><empty_output><jupyter_text>Now that I have the dataframe with total number of respondents who rated a particular cuisine, I use information on this dataframe to produce another dataframe with the percentage of respondents who rated a cuisine by the scale of 1 to 5.<jupyter_code>rate_percentage = [] # create an empty list contains the percentage

for country in countries:   # for each country in the country list specifed above
    
    # count each value (1 to 5) of each country column     # divide by the total of respodents in rate_total_df
    # round this number to 2 decimals for readability in our heatmap
    percentage = round(df[country].value_counts() / rate_total_df.loc[rate_total_df['Country'] == country]['Total'].unique(), 2)
    # when each percentage is caculated, append the value to the list
    rate_percentage.append(percentage)

# create a new dataframe with the list of percentage, reset index
rate_percentage = pd.DataFrame(rate_percentage).reset_index()
# take a look at what we just produced
rate_percentage.head()<jupyter_output><empty_output><jupyter_text>Now to make this dataframe long form to be compatible to Altair, we want to "melt" this dataframe so that all ratings (1 to 5) belong to one column, and the corresponding percentage values to another column. To do this, we use Pandas's pd.melt<jupyter_code># create a new dataframe
                             # specify dataframe to melt     # specify which column to keep as a guide
rate_percentage_melt = pd.melt(rate_percentage.reset_index(), id_vars=['index'],
                               # specify which columns to melt
                               # note: column names are normally strings, but in this case the my dataframe
                               # decided that these column names are float instead of string of float
                                value_vars = [1.0, 2.0, 3.0, 4.0, 5.0],
                               # name the column of the rating    # name the column to the value of rating
                                var_name='Rating', value_name='Percentage')
# take a look
rate_percentage_melt<jupyter_output><empty_output><jupyter_text>I now proceed to creating a heatmap using the above dataframe<jupyter_code># create a heatmap
chart = alt.Chart(rate_percentage_melt).mark_rect().encode(
    y=alt.Y('index:O',sort='ascending', # sort country name alphabetically
           axis=alt.Axis(labels=True, ticks=False, title=None)),
    x=alt.X('Rating:O', axis=alt.Axis(title=None, ticks=False)),
    color=alt.Color('Percentage:Q', scale=alt.Scale(type='linear', 
                                                    range=['#fde0dd', '#fcc5c0', '#fa9fb5', '#f768a1', 
                                                           '#dd3497', '#ae017e', '#7a0177', '#c51b8a']))
)
# create text that are the percentage to go into each rectange in the heatmap
text = alt.Chart(rate_percentage_melt).mark_text(baseline='middle').encode(
    y=alt.Y('index:O', sort='ascending'), # sort country name alphabetically
    x=alt.X('Rating:O'),
    text='Percentage:Q'
)
# add these two graph together
heat_map = (chart + text).properties(
    width=300,
    height=1000,
    title={'text': ["America's Favorite Cuisine"],
           'subtitle': ['Cuisine Favorability Ratings by',
                        'The Total Number of Respondents Who Rated that Cuisine']}

).configure_view(
    strokeWidth=0,
    
).configure_scale(
    bandPaddingInner=0.05

).configure_axis(domain=False, 
                 labelPadding=10,
                 
).configure_title(
    fontSize=20,
    fontWeight='bold',
    color='#49006a',
    subtitlePadding=7
    
)
heat_map<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08617870539578724, 'fraction_numerical': 0.007213619313263441, 'mean_word_length': 3.8839830906528885, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '440447', 'license': 'MIT', 'n_tokens_mistral': 2908, 'n_tokens_neox': 2765, 'n_words': 1163, 'path': 'Altair heatmap and long form data transformation.ipynb', 'repo_name': 'annettehan/America-s-favorite-cuisine-Altair', 'size': '10397'}