<jupyter_start><jupyter_text>Section 3: Data VisualizationThe human brain excels at finding patterns in visual representations of the data; so in this section, we will learn how to visualize data using pandas along with the Matplotlib and Seaborn libraries for additional features. We will create a variety of visualizations that will help us better understand our data.  Plotting with pandasWe can create a variety of visualizations using the `plot()` method. In this section, we will take a brief tour of some of this functionality, which under the hood uses Matplotlib. Once again, we will be working with the TSA traveler throughput data that we cleaned up in the previous section:<jupyter_code>import pandas as pd

tsa_melted_holiday_travel = pd.read_csv(
    '../data/tsa_melted_holiday_travel.csv', 
    parse_dates=True, index_col='date'
)
tsa_melted_holiday_travel.head()<jupyter_output><empty_output><jupyter_text>To embed our plots in the notebook, we will also call the `%matplotlib inline` magic:<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Line plotsThe `plot()` method will generate line plots for all numeric columns by default:<jupyter_code>tsa_melted_holiday_travel.drop(columns='year').loc['2020'].assign(
    **{
        '7D MA': lambda x: x.travelers.rolling('7D').mean(),
        'YTD mean': lambda x: x.travelers.expanding().mean()
      }
).plot(
    title='2020 TSA Traveler Throughput', 
    ylabel='travelers', alpha=0.8
)<jupyter_output><empty_output><jupyter_text>The `plot()` method returns an `Axes` object that can be modified further (e.g. to add reference lines, annotations, labels, etc.). Let's walk through an example.  Bar plotsFor our next example, we will plot vertical bars to compare monthly TSA traveler throughput across years. Let's start by creating a pivot table with the information we need:<jupyter_code>plot_data = tsa_melted_holiday_travel['2019':'2021-04'].assign(month=lambda x: x.index.month)\
    .pivot_table(index='month', columns='year', values='travelers', aggfunc='sum')
plot_data.head()<jupyter_output><empty_output><jupyter_text>Pandas offers other plot types via the `kind` parameter, so we specify `kind='bar'` when calling the `plot()` method. Then, we further format the visualization using the `Axes` object returned by the `plot()` method:<jupyter_code>import calendar
from matplotlib import ticker

ax = plot_data.plot(
    kind='bar', rot=0, xlabel='', ylabel='travelers',
    figsize=(10, 2), title='TSA Monthly Traveler Throughput'
)

# use month abbreviations for the ticks on the x-axis
ax.set_xticklabels(calendar.month_abbr[1:])

# show y-axis labels in millions instead of scientific notation
ax.yaxis.set_major_formatter(ticker.EngFormatter())

# customize the legend
ax.legend(title='', loc='center', bbox_to_anchor=(0.5, -0.3), ncol=3, frameon=False)<jupyter_output><empty_output><jupyter_text>Some additional things to keep in mind:- Matplotlib's `ticker` module provides functionality for customizing both the tick labels and locations &ndash; check out the [documentation](https://matplotlib.org/stable/api/ticker_api.html) for more information.- Pandas supports horizontal and stacked bars as well; [this](https://opendatascience.com/how-to-pivot-and-plot-data-with-pandas/) blog post shows how to make stacked horizontal bars using a pivot table.- The `plot()` method takes a lot of [parameters](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.html), many of which get passed down to Matplotlib; however, sometimes we need to use Matplotlib calls directly.  Plotting distributionsLet's now compare the distribution of daily TSA traveler throughput across years. We will create a subplot for each year with both a histogram and a kernel density estimate (KDE) of the distribution.  Pandas has generated the `Figure` and `Axes` objects for both examples so far, but we can build custom layouts by creating them ourselves with Matplotlib using the `plt.subplots()` function. First, we will need to import the `pyplot` module:<jupyter_code>import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>While pandas lets us specify that we want subplots and their layout (with the `subplots` and `layout` parameters, respectively), using Matplotlib to create the subplots directly gives us additional flexibility:<jupyter_code># define the subplot layout
fig, axes = plt.subplots(3, 1, sharex=True, sharey=True, figsize=(8, 4))

for year, ax in zip(tsa_melted_holiday_travel.year.unique(), axes):
    plot_data = tsa_melted_holiday_travel.loc[str(year)].travelers
    plot_data.plot(kind='hist', legend=False, density=True, alpha=0.8, ax=ax)
    plot_data.plot(kind='kde', legend=False, color='blue', ax=ax)
    ax.set(title=f'{year} TSA Traveler Throughput', xlabel='travelers')

fig.tight_layout() # handle overlaps<jupyter_output><empty_output><jupyter_text>*Tip: If you're new to `zip()`, check out [this](https://realpython.com/python-zip-function/) article.*  Plotting with SeabornThe Seaborn library provides the means to easily visualize long-format data without first pivoting it. In addition, it also offers some additional plot types &ndash; once again building on top of Mtplotlib. Here, we will look at a few examples of visualizations we can create with Seaborn.  Visualizing long-format dataWith Seaborn, we can specify plot colors according to values of a column with the `hue` parameter. When working with functions that generate subplots, we can also specify how to split the subplots by values of a long-format column with the `col` and `row` parameters. Here, we revisit the comparison of the distribution of TSA traveler throughput across years:<jupyter_code>import seaborn as sns

sns.displot(data=tsa_melted_holiday_travel, x='travelers', kde=True, col='year')<jupyter_output><empty_output><jupyter_text>HeatmapsWe can use Seaborn to visualize pivot tables as heatmaps:<jupyter_code>data = tsa_melted_holiday_travel['2019':'2021-04'].assign(month=lambda x: x.index.month)\
    .pivot_table(index='month', columns='year', values='travelers', aggfunc='sum')

ax = sns.heatmap(data=data / 1e6, cmap='Blues', annot=True, fmt='.1f')
ax.set_yticklabels(calendar.month_abbr[1:], rotation=0)
ax.set_title('Total TSA Traveler Throughput (in millions)')<jupyter_output><empty_output><jupyter_text>We're moving on from Seaborn now, but there is a lot more available in the [API](https://seaborn.pydata.org/api.html). Be sure to check out the following at a minimum:- [pairwise plots](https://seaborn.pydata.org/generated/seaborn.pairplot.htmlseaborn.pairplot) with `pairplot()`- [categorical scatter plots](https://seaborn.pydata.org/generated/seaborn.swarmplot.htmlseaborn.swarmplot) with `swarmplot()`- [joint distribution plots](https://seaborn.pydata.org/generated/seaborn.jointplot.htmlseaborn.jointplot) with `jointplot()`- [FacetGrids](https://seaborn.pydata.org/generated/seaborn.FacetGrid.htmlseaborn.FacetGrid) for custom layouts with any plot type  Customizing plots with MatplotlibIn this final section, we will discuss how to use Matplotlib to customize plots. Since there is a lot of functionality available, we will only be covering how to add shaded regions and annotations here, but be sure to check out the [documentation](https://matplotlib.org/) for more.  Adding shaded regionsWhen looking at a plot of TSA traveler throughput over time, it's helpful to indicate periods during which there was holiday travel. We can do so with the `axvspan()` method:<jupyter_code>plot_data = tsa_melted_holiday_travel['2019-05':'2019-11']
ax = plot_data.travelers.plot(
    title='TSA Traveler Throughput', ylabel='travelers', figsize=(12, 2)
)
ax.yaxis.set_major_formatter(ticker.EngFormatter())

# collect the holiday ranges (start and end dates)
holiday_ranges = plot_data.dropna().reset_index()\
    .groupby('holiday').agg({'date': ['min', 'max']})

# create shaded regions for each holiday in the plot
for start_date, end_date in holiday_ranges.to_numpy():
    ax.axvspan(start_date, end_date, color='gray', alpha=0.2)<jupyter_output><empty_output><jupyter_text>*Tip: Use `axhspan()` for horizontally shaded regions and `axvline()` / `axhline()` for vertical/horizontal reference lines.*  Adding annotationsWe can use the `annotate()` method to add annotations to the plot. Here, we point out the day in 2019 with the highest TSA traveler throughput, which was the day after Thanksgiving:<jupyter_code>plot_data = tsa_melted_holiday_travel.loc['2019']
ax = plot_data.travelers.plot(
    title='TSA Traveler Throughput', ylabel='travelers', figsize=(12, 2)
)
ax.yaxis.set_major_formatter(ticker.EngFormatter())

# highest throughput
max_throughput_date = plot_data.travelers.idxmax()
max_throughput = plot_data.travelers.max()
_ = ax.annotate(
    f'{max_throughput_date:%b %d}\n({max_throughput / 1e6:.2f} M)',
    xy=(max_throughput_date, max_throughput),
    xytext=(max_throughput_date - pd.Timedelta(days=23), max_throughput * 0.9),
    arrowprops={'arrowstyle': '->'}, ha='center'
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10430863926137612, 'fraction_numerical': 0.009782369751593757, 'mean_word_length': 5.988479262672811, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 46, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '235429', 'license': 'MIT', 'n_tokens_mistral': 2943, 'n_tokens_neox': 2750, 'n_words': 917, 'path': 'notebooks/3-data_visualization.ipynb', 'repo_name': 'sebastian2296/pandas-workshop', 'size': '9098'}