<jupyter_start><jupyter_text>Visualizing Multiple Data Sets - Part 2: Histograms for Partitioned Data Now we return to our Covid data set and apply what we have learned about partitions and summary statistics to set up what we call binary comparisons: these are comparisons between two groups. As before, we begin by importing the data from the CSV file:<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

df = pd.read_csv(
    "https://raw.githubusercontent.com/jmshea/Foundations-of-Data-Science-with-Python/main/03-first-data/covid-merged.csv"
)
df.set_index("state", inplace=True)
df["cases_norm"] = df["cases"] / df["population"] * 1000<jupyter_output><empty_output><jupyter_text>In the following subsections, we will partition the data based on normalized gdp and how urban the state’s population is. In each case, we will partition the data into two sets by comparing the metric being studied to the median value of that metric. Here, we have chosen median because it will result in the data set being partitioned into subsets that are approximately equal in size.  Partitioning Based on GDPThe gross domestic product (GDP) provides a measure of how affluent a state is. As we expect that GDP will increase along with population, we first add normalized GDP (in millions) per capita (in thousands) to our dataframe[^per-capita].[^per-capita]: Since both Covid cases and GDP are being normalized per capita, the normalization could actually be dropped. However, we keep it here to be consistent with the analysis of the effect of the urban index.<jupyter_code>df["gdp_norm"] = df["gdp"] / df["population"] * 1000<jupyter_output><empty_output><jupyter_text>We will partition the data around the median of the normalized GDP, which is<jupyter_code>m_gdp = df["gdp_norm"].median()
m_gdp<jupyter_output><empty_output><jupyter_text>To partition the data let's first see what happens when a numerical column of a dataframe is compared to a value:<jupyter_code>df["gdp_norm"] > m_gdp<jupyter_output><empty_output><jupyter_text>Pandas returns a series that contains Boolean values indicating whether the comparison for the value at each index was true or false. We can use such a series to partition our dataframe if we pass it as the index (in square brackets) to the data frame. So, we can create a dataframe with states that have population higher than the median as follows:<jupyter_code>higher_gdp = df[df["gdp_norm"] > m_gdp]
higher_gdp<jupyter_output><empty_output><jupyter_text>Note the exceptionally high normalized case rate for New York (15.92) and New Jersey (13.4).  By changine the inequality to $\le$, we can create a dataframe with states at or below the median population:<jupyter_code>lower_gdp = df[df["gdp_norm"] <= m_gdp]
lower_gdp<jupyter_output><empty_output><jupyter_text>As expected, the two dataframes have the same size:<jupyter_code>len(higher_gdp), len(lower_gdp)<jupyter_output><empty_output><jupyter_text>Let's generate histograms for case data for both the higher GDP and lower GDP data sets. We plot them on the same axes to make it easier to compare them:<jupyter_code>plt.hist(higher_gdp["cases_norm"], alpha=0.7, label="GDP/pop > median")
plt.hist(lower_gdp["cases_norm"], alpha=0.7, label="GDP/pop <= median")

plt.legend()
plt.xlabel("Cases per population in thousands")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>An issue that arises when plotting histograms on the same axes is that it becomes hard to compare values when the histograms use different bins. For example, from about 0.5 to 1.8, the histogram for the higher GDP group has a single bar with 13 entries. On the other hand, the histogram for the lower GDP group has two bars, but together those two bars sum up to 19. To resolve this issue, lets just pass bins that span the region as the bins for each histogram:<jupyter_code>mybins = np.arange(0, 18, 2)
mybins
plt.hist(higher_gdp["cases_norm"], alpha=0.7, label="GDP/pop > median", bins=mybins)
plt.hist(lower_gdp["cases_norm"], alpha=0.7, label="GDP/pop <= median", bins=mybins)

plt.legend()
plt.xlabel("Cases per population in thousands")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>The histogram shows that states with GDPs over the median have higher case counts in general (for instance, the larger bars in the range 2-5), as well as larger maximum values (above 8).  One reason for this might be that states with higher GDPs may have network effects, where more of the population interacts with a larger number of people. For instance, such states may have more of their population employed in office jobs and likely to travel for their work.  When an effect is observed visually, we usually want to quantify that effect in some way and then test whether any observed difference is "real" (rather than just caused by randomness). Let's compare the average of the normalized number of cases for these two different groups:<jupyter_code>hgdp_mu = higher_gdp["cases_norm"].mean()
lgdp_mu = lower_gdp["cases_norm"].mean()
hgdp_mu, lgdp_mu<jupyter_output><empty_output><jupyter_text>There is a substantial difference between the two average values. However, the average is more sensitive to extreme values, such as the large normalized case rates of New York and New Jersey. One way to deal with this is to use a statistic that is not as sensitive to extrema; for example, the median is such a statistic:<jupyter_code>higher_gdp["cases_norm"].median(), lower_gdp["cases_norm"].median()<jupyter_output><empty_output><jupyter_text>**Exercise**A new data frame can be created that excludes New York and New Jersey using the following command:<jupyter_code>df2 = df.drop(["New York", "New Jersey"])<jupyter_output><empty_output><jupyter_text>Using this reduced data set, partition the data around the median normalized GDP, plot the histograms for the normalized Covid rates of the two partitions, and find the average normalized Covid rates for the two partitions. **Exercise**Partition the data around the **mean** GDP. Plot the histograms and find the averages for this new grouping.  Partitioning Based on Urban IndexSome of the networking effects mentioned in the previous section may be associated with populations living in large, urban areas. In this section, we partition the data based on the urban index, which is the percentage of the population that is living in an urban area.  As before, we start by finding the median urban index for the states:<jupyter_code>m_urban = df["urban"].median()
m_urban<jupyter_output><empty_output><jupyter_text>Now we use that median to partition the data and plot the histograms:<jupyter_code>higher_urban = df[df["urban"] > m_urban]
lower_urban = df[df["urban"] <= m_urban]
plt.hist(
    higher_urban["cases_norm"], alpha=0.7, label="Urban index > median", bins=mybins
)
plt.hist(
    lower_urban["cases_norm"], alpha=0.7, label="Urban index <= median", bins=mybins
)
plt.legend()
plt.xlabel("Cases per population in thousands")
plt.ylabel("Count")<jupyter_output><empty_output><jupyter_text>As with normalized GDP, a substantial difference in distribution of normalized number of Covid cases is seen for states that are more urban versus less urban. The average normalized case rates for these partitions are:<jupyter_code>higher_urban["cases_norm"].mean(), lower_urban["cases_norm"].mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07873376623376624, 'fraction_numerical': 0.006222943722943723, 'mean_word_length': 5.948308270676692, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 67, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '610632', 'license': 'FSFAP', 'n_tokens_mistral': 2230, 'n_tokens_neox': 2065, 'n_words': 917, 'path': '03-first-data/histogram.ipynb', 'repo_name': 'jmshea/intro-data-science-for-engineers', 'size': '7392'}