<jupyter_start><jupyter_text>INFO 3402 – Week 05: Histograms and Perception[<NAME>, Ph.D.](http://brianckeegan.com/)  [Assistant Professor, Department of Information Science](https://www.colorado.edu/cmci/people/information-science/brian-c-keegan)  University of Colorado Boulder  Copyright and distributed under an [MIT License](https://opensource.org/licenses/MIT)    Setup<jupyter_code>import numpy as np
np.set_printoptions(suppress=True)
import pandas as pd

idx = pd.IndexSlice

pd.options.display.max_columns = 100<jupyter_output><empty_output><jupyter_text>Load the Boulder and Broomfield weather data from Week 1.<jupyter_code>boulder_df = pd.read_csv('../Week 01 - Loading and Documentation/boulder_weather.csv',
                         header=1,
                         sep='|'
                        )
boulder_df.head()<jupyter_output><empty_output><jupyter_text>BackgroundWe've discussed the *shape* of DataFrames in terms of rows, columns, MultiIndicies, and tidy data. But for our first encounter with data visualization, we want to explore the *shape* of data values themselves. A classic and still influential practice is to visualize the distribution of data values as histograms.We are going to spend a lot of time for the rest of class working with the [matplotlib](https://matplotlib.org/) data visualization library. Many other parts of Python's data analysis ecosystem, including pandas and seaborn, interface with matplotlib for visualization. In particular, we are going to use matplotlib's "[object-oriented](https://matplotlib.org/stable/tutorials/introductory/usage.html?highlight=object%20orientedcoding-styles)" coding style for constucting and customizing visualizations. Here is the "anatomy" of a matplotlib figure with the corresponding methods for changing that part of a figure. We'll be referencing this a lot!![Anatomy of a matplotlib figure](https://matplotlib.org/stable/_images/sphx_glr_anatomy_001.png)  Setting up matplotlibWhen we use matplotlib within a Jupyter Notebook, we have to run a special command called a "[cell magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html)" so that the images it generates are included in the notebook. If you didn't have the `%matplotlib inline` cell magic, you would need to run a command like `plt.show()` or `plt.savefig()` to see the output of your visualization. We also construct and manage plots through matplotlib's `pyplot` ([docs](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.html)) interface. The `pyplot` module contains many functions for improving plots.**Take-away 01**: If you're using matplotlib in a Jupyter Notebook, include the `%matplotlib inline` command.  **Take-away 02**: Always import and alias the pyplot interface at a minimum: `import matplotlib.pyplot as plt`<jupyter_code>%matplotlib inline

import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Exercise 01: Load Broomfield weather data<jupyter_code>broomfield_df = pd.read_csv('../Week 01 - Loading and Documentation/broomfield_weather.csv',
                         header=1,
                         sep='|'
                        )
broomfield_df.head()<jupyter_output><empty_output><jupyter_text>Creating and customizing figures and subplotsThis is our first stop on the "object-oriented" coding style in matplotlib. Before we do anything, we just want to create a figure called `f` containing a single subplot called `ax`. These are empty until we fill them.<jupyter_code>f, ax = plt.subplots()<jupyter_output><empty_output><jupyter_text>We can call the `f` Figure object and it will return the (now-empty) visualization.<jupyter_code>f<jupyter_output><empty_output><jupyter_text>The `f` object contains an enormous number of methods and attributes, many of which we will explore in the weeks to come. The `.get_axes()` method retrieves the subplot objects (called axes) within the figure object. There's only one subplot in this figure, so it's a list of one ax. Later, we will see how to make figures with multiple subplots (or see [this tutorial](https://matplotlib.org/stable/tutorials/intermediate/arranging_axes.html) if you can't wait).<jupyter_code>f.get_axes()<jupyter_output><empty_output><jupyter_text>The `ax` object (or the `AxesSubplot:` object within the figure's list of axes) have their own methods and attributes. The `.get_children()` shows all the components of this "empty" plot: * `spines` for the four borders (top, bottom, right, left)* the `XAxis` and `YAxis` with the tick labels 0.0 to 1.0* empty `Text` objects for the xlabel, ylabel, and title* a `Rectangle` patch object that overlaps with the spines.<jupyter_code>ax.get_children()<jupyter_output><empty_output><jupyter_text>The rectangle object is a box that has a height and width of 1 and starts at the origin (0,0).<jupyter_code>rect = ax.get_children()[-1]
rect.get_height(), rect.get_width(), rect.get_xy()<jupyter_output><empty_output><jupyter_text>We could mutate this Rectangle: color it red and give it a new width and height.<jupyter_code>rect.set_facecolor('red')
rect.set_height(.5)
rect.set_width(.5)<jupyter_output><empty_output><jupyter_text>If we check the `f` object again, we can see this rectangle within the axes object within the figure object is now red and has half the height and width as before.<jupyter_code>f<jupyter_output><empty_output><jupyter_text>Exercise 02: Mutate a patch edgecolor and linewidthMutate the `rect` object ([docs](https://matplotlib.org/stable/api/_as_gen/matplotlib.patches.Rectangle.html)) so that it has an "edgecolor" of green and "linewidth" of 10. Show the result in `f`.<jupyter_code>rect.set_edgecolor('green')
rect.set_linewidth(10)
rect.set_linestyle('--')
f<jupyter_output><empty_output><jupyter_text>One-dimensional histogramWe're going to make a simple histogram using the weather data we loaded above.First, retrieve the values of the Boulder "TEMP" Series as a numpy `array` using the `.values` attribute and store as `boulder_temps`.<jupyter_code>boulder_temps = boulder_df['TEMP'].values
boulder_temps<jupyter_output><empty_output><jupyter_text>Now make an empty figure and subplot.<jupyter_code>f,ax = plt.subplots()<jupyter_output><empty_output><jupyter_text>Now plot a histogram of `boulder_temps` in the `ax` object. The `.hist` method returns three values:* The count of values in each bin* The bounadies of each bin* The `BarContainer` object that populates/replaces the contents of the empty `ax`<jupyter_code>ax.hist(boulder_temps)<jupyter_output><empty_output><jupyter_text>But where's the plot? Check `f`.<jupyter_code>f<jupyter_output><empty_output><jupyter_text>If you do the create the figure and subplot and make the histogram in the same cell (which we'll do from now on) the figure appears automatically. The other output from the `hist` method is still included: if you want to silence this, add a semicolon to the end of the line.<jupyter_code>f,ax = plt.subplots()
ax.hist(boulder_temps); # Silence the bin counts and boundaries<jupyter_output><empty_output><jupyter_text>Number of binsWe can also change the number of bins in the histogram with the "bins" parameter. Start with few bins.<jupyter_code>f,ax = plt.subplots()
ax.hist(boulder_temps,bins=5);<jupyter_output><empty_output><jupyter_text>Try a large number of bins. We start to see there might be two-ish peaks to the data, which would make sense for different temperatures in the winter and summer.<jupyter_code>f,ax = plt.subplots()
ax.hist(boulder_temps,bins=50);<jupyter_output><empty_output><jupyter_text>Try a really big number of bins. The bimodal distribution is perhaps clearer here, but this probably more bins than intended for a histogram.<jupyter_code>f,ax = plt.subplots()
ax.hist(boulder_temps,bins=500)

ax.set_xlabel('Temperature (F)')
ax.set_ylabel('Count (Days)')
ax.set_title('Temperatures in Boulder, Colorado (2010-2021)')

f.savefig('too_many_bins.png',facecolor='w')<jupyter_output><empty_output><jupyter_text>Label axesOne of the most essential visualization practices is to label our axes. Add an x-label and y-label.<jupyter_code>f,ax = plt.subplots()
ax.hist(boulder_temps,bins=25)

ax.set_xlabel('Temperature (F)')
ax.set_ylabel('Count (Days)')
ax.set_title('Temperatures in Boulder, Colorado (2010-2021)')<jupyter_output><empty_output><jupyter_text>Exercise 03: Plot a histogram of Broomfield's temperaturesCreate an empty figure and subplot, extract the "TEMP" values from `broomfield_df`, make and display a histogram, and label your axes.<jupyter_code>f,ax = plt.subplots()

ax.hist(broomfield_df['TEMP'],bins=25);
f,a_subplot = plt.subplots()

broomfield_df['TEMP'].plot.hist(ax=a_subplot,bins=50)<jupyter_output><empty_output><jupyter_text>Skewed dataThe weather data has nice statistical and visual properties of being (approximately) normally distributed. Many kinds of data do not have this property and instead show a strong skew with low counts being significantly more represented than high counts.Load in the county COVID data from Week 3 as `co_covid_df`.<jupyter_code>co_covid_df = pd.read_csv('../Week 03 - Combining and Validation/co_county_covid.csv')
co_covid_df.head()<jupyter_output><empty_output><jupyter_text>Extract the values for the "Cases" column and make a histogram. There's an enormous number of county-months where there are 0 or few cases and a handful of county-months where there are thousands of cases. This is not an informative data visualization.<jupyter_code>covid_cases = co_covid_df['Cases'].values

f,ax = plt.subplots()
ax.hist(covid_cases)

ax.set_xlabel('Cases')
ax.set_ylabel('Count (Monthly)')
ax.set_title('COVID-19 Cases in Colorado Counties')

f.savefig('county_cases.png',facecolor='w')<jupyter_output><empty_output><jupyter_text>Even adding more bins does not improve this visualization.<jupyter_code>f,ax = plt.subplots()
ax.hist(covid_cases,bins=50)

ax.set_xlabel('Cases')
ax.set_ylabel('Count (Monthly)')
ax.set_title('COVID-19 Cases in Colorado Counties')<jupyter_output><empty_output><jupyter_text>A common approach to handling skewed data like this is to use a log-transformation. Rather than the bin sizes being linear (1 to 10, 11 to 20, *etc*.) the bin sizes are exponential (1 to 10, 11 to 100, 101 to 1000, *etc*.). A numpy function called `linspace` ([docs](https://numpy.org/doc/stable/reference/generated/numpy.linspace.html)) can illustrate creating 10 linearly-spaced bins from 1 to 1000 and a similar function called `geomspace` ([docs](https://numpy.org/doc/stable/reference/generated/numpy.geomspace.html)) to create 10 exponentially-spaced bins from 1 to 1000.<jupyter_code>np.linspace(1,10000,11)
np.geomspace(1,10000,11) # The start and stop values are in base units 10^0 = 1, 10^4 = 10000<jupyter_output><empty_output><jupyter_text>We can pass these bin boundaries into the "bins" parameter for `hist`.<jupyter_code>f,ax = plt.subplots()
ax.hist(covid_cases,bins=np.linspace(1,10000,11)) # Linear bin sizes

ax.set_xlabel('Cases')
ax.set_ylabel('Count (Monthly)')
ax.set_title('COVID-19 Cases in Colorado Counties')<jupyter_output><empty_output><jupyter_text>Use the geometrically-spaced bins instead. There's definitely something different going on here!<jupyter_code>f,ax = plt.subplots()
ax.hist(covid_cases,bins=np.geomspace(1,10000,11)) # Geometric bin sizes

ax.set_xlabel('Cases')
ax.set_ylabel('Count (Monthly)')
ax.set_title('COVID-19 Cases in Colorado Counties')<jupyter_output><empty_output><jupyter_text>Because we're using geometric instead of linear bins, we should also use a geometric scale for the x and y-axes instead of a linear scale. Use the `.set_xscale()` ([docs](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.set_xscale.html)) method to turn the x-axis ticks into a geometric rather than linear progression. This is called a "semi-log" plot.You could also make the y-axis have a geometric/logarithmic progression (uncomment the last line with `.set_yscale('log')`. This would be a "log-log" plot.<jupyter_code>f,ax = plt.subplots()
ax.hist(covid_cases,bins=np.geomspace(1,10000,11)) # Geometric bin sizes

ax.set_xlabel('Cases')
ax.set_ylabel('Count (Monthly)')
ax.set_title('Monthly COVID-19 Cases in Colorado Counties')

ax.set_xscale('log')
# ax.set_yscale('log')<jupyter_output><empty_output><jupyter_text>Advanced: Why should I care about log-binned data?(This won't be on the quiz, but if you're curious, read on)Make some data following different distributions: [exponential](https://numpy.org/doc/stable/reference/random/generated/numpy.random.exponential.html), [lognormal](https://numpy.org/doc/stable/reference/random/generated/numpy.random.lognormal.html), [pareto](https://numpy.org/doc/stable/reference/random/generated/numpy.random.pareto.html), and [power](https://numpy.org/doc/stable/reference/random/generated/numpy.random.power.html).<jupyter_code># https://stackoverflow.com/a/58331860/1574687

# Rescale data to be (0,1)
from sklearn.preprocessing import minmax_scale
_size = 1000

# Make random distributions of different shapes
random_exponential = minmax_scale(np.random.exponential(size=_size),feature_range=(0,1))
random_lognormal = minmax_scale(np.random.lognormal(size=_size),feature_range=(0,1))
random_pareto = minmax_scale(np.random.pareto(.9,size=_size),feature_range=(0,1))
random_power = minmax_scale(np.random.power(.1,size=_size),feature_range=(0,1))<jupyter_output><empty_output><jupyter_text>On linear axes and bins these distributions look pretty similar.<jupyter_code># Make four subplots
f,(ax0,ax1,ax2,ax3) = plt.subplots(
    nrows = 1,
    ncols = 4,
    figsize=(16,4)
)

# Histograms for each subplot
ax0.hist(random_exponential,bins=25)
ax1.hist(random_lognormal,bins=25)
ax2.hist(random_pareto,bins=25)
ax3.hist(random_power,bins=25)

# Set titles
ax0.set_title('Exponential')
ax1.set_title('Log-normal')
ax2.set_title('Pareto')
ax3.set_title('Power')

# Super-title and cleanup
f.suptitle('Long-tailed distributions')
f.set_facecolor('w')
f.tight_layout()<jupyter_output><empty_output><jupyter_text>On log axes and bins they look very different!<jupyter_code># Make four subplots
f,(ax0,ax1,ax2,ax3) = plt.subplots(
    nrows = 1,
    ncols = 4,
    figsize = (16,4),
    subplot_kw = {'xscale':'log'}
)

# Histograms for each subplot with geomspace bins
ax0.hist(random_exponential,bins=np.geomspace(1e-3,1e0,25))
ax1.hist(random_lognormal,bins=np.geomspace(1e-3,1e0,25))
ax2.hist(random_pareto,bins=np.geomspace(1e-4,1e0,25))
ax3.hist(random_power,bins=np.geomspace(1e-3,1e0,25))

# Set titles
ax0.set_title('Exponential')
ax1.set_title('Log-normal')
ax2.set_title('Pareto')
ax3.set_title('Power')

# Super-title and cleanup
f.suptitle('Long-tailed distributions')
f.set_facecolor('w')
f.tight_layout()<jupyter_output><empty_output><jupyter_text>Exercise 04: Make a log-log histogram of deathsCreate an empty figure and subplot, extract the "Deaths" values from `co_covid_df`, make a histogram with an appropriate number and size of bins, label your axes, and rescale the axes.<jupyter_code>co_covid_df.head()
f,covid_deaths_ax = plt.subplots()

co_covid_df['Deaths'].plot.hist(ax=covid_deaths_ax)
no_neg_deaths = co_covid_df[co_covid_df['Deaths'] > 0]
f,ax = plt.subplots()
no_neg_deaths['Deaths'].plot.hist(ax=ax,bins=np.geomspace(1,40,10))
ax.set_xscale('log')<jupyter_output><empty_output><jupyter_text>Visualizing multiple histograms([Adapted from this gallery](https://matplotlib.org/stable/gallery/statistics/histogram_multihist.html))We may want to visualize more than one histogram at once. Let's work with the "TEMP" data from the Boulder and Broomfield data. These data are different sizes reflecting different dates, start by inner joining them together so they are the same size. I'm using the "suffixes" parameter because the column names are otherwise identical (tidy data violation!).<jupyter_code>boulder_df.shape, broomfield_df.shape
both_df = pd.merge(
    left = boulder_df,
    right = broomfield_df,
    left_on = 'DATE',
    right_on = 'DATE',
    how = 'inner',
    suffixes = ('_BOULDER','_BROOMFIELD')
)

both_df.head()
boulder_temps = both_df['TEMP_BOULDER'].values
broomfield_temps = both_df['TEMP_BROOMFIELD'].values<jupyter_output><empty_output><jupyter_text>Option 1 is to make side-by-side plots. The "ncols" parameter is changed from default 1 to 2. Now the `subplots` function produces two subplot axes that I capture as `ax0` and `ax1`.<jupyter_code>f,(ax0,ax1) = plt.subplots(1,2)
f,(ax0,ax1) = plt.subplots(
    nrows=1,
    ncols=2,
    figsize=(8,4),
    sharey=True
)

ax0.hist(boulder_temps,bins=25)
ax1.hist(broomfield_temps,bins=25)

ax0.set_xlabel('Temperature (F)')
ax0.set_ylabel('Count (Days)')
ax0.set_title('Boulder, Colorado')

ax1.set_xlabel('Temperature (F)')
ax1.set_ylabel('Count (Days)')
ax1.set_title('Broomfield, Colorado')

f.suptitle('Temperatures 2010-2021',fontweight='heavy')
f.tight_layout()

f.savefig('hist_temp_boulder_broomfield.png',facecolor='w')<jupyter_output><empty_output><jupyter_text>Option 2 is to make the histogram on a single plot but with different bars for each city. This is hard with default matplotlib, but is easier with pandas and seaborn.stack ([docs](https://numpy.org/doc/stable/reference/generated/numpy.stack.html)) the two arrays together (similar to a concat).<jupyter_code># Stack the numpy arrays together
stacked_arrays = np.stack([boulder_temps,broomfield_temps],axis=1)
boulder_temps.shape, broomfield_temps.shape, stacked_arrays.shape
f,ax = plt.subplots()

ax.hist(
    stacked_arrays,
    bins = 15,
    histtype = 'bar', 
    color = ['tab:red','tab:blue'], 
    label = ['Boulder','Broomfield']
)

ax.set_xlabel('Temperature (F)')
ax.set_ylabel('Count (Days)')
ax.set_title('Temperatures (2010-2021)',fontweight='heavy')
ax.legend(prop={'size': 10})<jupyter_output><empty_output><jupyter_text>Option 3 is to stack the bars on top of each other. This is perceptually hard to compare to identify differences in many cases.<jupyter_code>f,ax = plt.subplots()

ax.hist(
    stacked_arrays,
    bins = 15,
    histtype = 'bar', 
    stacked = True,
    color = ['tab:red','tab:blue'], 
    label = ['Boulder','Broomfield']
)

ax.set_xlabel('Temperature (F)')
ax.set_ylabel('Count (Days)')
ax.set_title('Temperatures (2010-2021)',fontweight='heavy')
ax.legend(prop={'size': 10})<jupyter_output><empty_output><jupyter_text>Plotting a cumulative histogram.<jupyter_code>f,ax = plt.subplots()

ax.hist(
    stacked_arrays,
    bins = 15,
    histtype = 'bar', 
    cumulative = True,
    color = ['tab:red','tab:blue'], 
    label = ['Boulder','Broomfield']
)

ax.set_xlabel('Temperature (F)')
ax.set_ylabel('Count (Days, cuml.)')
ax.set_title('Temperatures (2010-2021)',fontweight='heavy')
ax.legend(prop={'size': 10})<jupyter_output><empty_output><jupyter_text>Making multiple plots with pandas.<jupyter_code>co_covid_df.head()
co_covid_df['County'].unique()
13*5
axs.shape
co_covid_df.head()
f,axs = plt.subplots(nrows=13,ncols=5,figsize=(10,20))

co_covid_df.plot.hist(
    column='Cases',
    by=['County'],
    ax=axs,
    legend=False,
#     bins=np.geomspace(1,5000,10)
);

f.tight_layout()<jupyter_output><empty_output><jupyter_text>pandas methodsEverything we were doing above involved an extra step to extract data in pandas as numpy arrays and some new functions to manipulate those numpy arrays. Let's stay in pandas-land!You can call the `.hist()` method ([docs](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.hist.html)) on a pandas Series and it will return a figure and subplot with the histogram. There are some stylistic differences, but this is much easier!<jupyter_code>both_df['TEMP_BOULDER'].hist()<jupyter_output><empty_output><jupyter_text>You could also call `.hist()` on a DataFrame and get a histogram for each.<jupyter_code>both_df[['TEMP_BOULDER','TEMP_BROOMFIELD']].hist()<jupyter_output><empty_output><jupyter_text>The `.plot.hist()` method is distinct and will draw stacked, cumulative, horizontal and others.<jupyter_code>both_df[['TEMP_BOULDER','TEMP_BROOMFIELD']].plot.hist()
both_df[['TEMP_BOULDER','TEMP_BROOMFIELD']].plot.hist(stacked=True)
both_df[['TEMP_BOULDER','TEMP_BROOMFIELD']].plot.hist(orientation='horizontal')<jupyter_output><empty_output><jupyter_text>You can also pass pandas's subplots back to a figure you've already setup with `subplots`. Just pass the axes created by subplots to the "ax" parameter inside the pandas `.hist()` method.<jupyter_code>f,(ax0,ax1) = plt.subplots(ncols=2,figsize=(8,4))

both_df[['TEMP_BOULDER','TEMP_BROOMFIELD']].plot.hist(stacked=True,ax=ax0,legend=False)
both_df[['TEMP_BOULDER','TEMP_BROOMFIELD']].plot.hist(orientation='horizontal',ax=ax1)<jupyter_output><empty_output><jupyter_text>Exercise 05: Make a histogram of COVID cases and deaths<jupyter_code>no_neg_deaths.head(2)
f,(ax0,ax1) = plt.subplots(1,2)

no_neg_deaths['Cases'].plot.hist(ax=ax0)
no_neg_deaths['Deaths'].plot.hist(ax=ax1)

# ax0.set_xscale('log')
# ax1.set_xscale('log')
no_neg_deaths['Death'].cumsum()
f,(ax0,ax1) = plt.subplots(1,2)

no_neg_deaths['Cases'].plot.hist(ax=ax0,bins=np.geomspace(1,10000,10))
no_neg_deaths['Deaths'].plot.hist(ax=ax1,bins=np.geomspace(1,100,10))

ax0.set_xscale('log')
ax1.set_xscale('log')<jupyter_output><empty_output><jupyter_text>Advanced: Two-dimensional histogramYou can visualize the counts of two series of data. This is a bit like a terrain map, darker colors are lower and brighter colors are higher. Basically, how many days did Boulder and Broomfield have these temperature values? Unsurprisingly, two nearby cities have similar temperatures: all the values lie along the diagonal, there are no days when it's 80 in Boulder and 20 in Broomfield.<jupyter_code>f,ax = plt.subplots(figsize=(7,6))

counts, xedges, yedges, im = ax.hist2d(
    x = boulder_temps,
    y = broomfield_temps,
    bins = 25
)

ax.set_xlabel('Temperatures in Boulder (F)')
ax.set_ylabel('Temperatures in Broomfield (F)')
f.colorbar(im,ax=ax)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11792238480414433, 'fraction_numerical': 0.019176588203217305, 'mean_word_length': 6.258245382585752, 'pattern_counts': {'":': 0, '<': 182, '<?xml version=': 0, '>': 183, 'https://': 20, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '513793', 'license': 'MIT', 'n_tokens_mistral': 7936, 'n_tokens_neox': 7419, 'n_words': 1954, 'path': 'Week 05 - Histograms/Week 05 - Lecture.ipynb', 'repo_name': 'cuinfoscience/INFO3402-Spring2022', 'size': '22015'}