<jupyter_start><jupyter_text>Exploring Data in NotebooksThe two most common types of data encountered in typical data science workflows are tabular (columnar) datasets and raster (array) data. In the Python ecosystem, the former type of data has gradually standardized around the pandas [DataFrame API](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) while raster data has standarized around the [NumPy API](https://numpy.org/).Examples of libraries offering `DataFrame` style objects include [Dask](https://dask.org/), [Rapids](https://rapids.ai/), [GeoPandas](https:geopandas.org/), [Streamz](https://streamz.readthedocs.io/) and of course [Pandas](https:pandas.pydata.org) itself. For array data, you can use [XArray](http://xarray.pydata.org/) to specify labelled multidimensional arrays or [NumPy](https://numpy.org/) ndarrays.This notebook is available as part of an `anaconda project` archive that you can download [here](https://anaconda.org/jlstevens/project/exploring-data). If you extract the archive, you will also find a notebook called `Reproducibly_Capturing_Code.ipynb` which details the steps needed to set up the  environment necessary to run this notebook. More information can be found in the *Capturing your Python code as a reproducible, deployable project* companion talk.In this notebook, we will see how [hvplot](https://hvplot.holoviz.org/) allows you to visualize data in all these various formats using a `.plot` style API inspired by [pandas](https://pandas.pydata.org/). Our first step is therefore to take a look at some simple examples of what you can do with `.plot` on a pandas `DataFrame` without `hvplot`, starting with a pandas import:<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>The data we will examine lists the number of [cases of measles and pertussis](http://graphics.wsj.com/infectious-diseases-and-vaccines/b02g20t20w15) (per 100,000 people) over time in each US state from 1928 to 2011:<jupyter_code>df = pd.read_csv('diseases.csv.gz')
df.head()<jupyter_output><empty_output><jupyter_text>The `DataFrame` named `df` has a `.plot` method we can simply call after running the `%matplotlib inline` notebook magic:<jupyter_code>%matplotlib inline
df.plot();<jupyter_output><empty_output><jupyter_text>At this point we can note two things about the plot above:* This plot is rendered as a static image with [matplotlib](https://matplotlib.org/) which means it is not interactive.* Without any specification from the user, the `.plot` call renders a plot that displays all the available data but this plot is hard to interpret.Using the `numpy.sum` function, we can build a new `DataFrame` indexed by `'Year'` that has a `'measles'` column that is the aggregate over that year:<jupyter_code>import numpy as np

by_year = df[["Year","measles"]].groupby("Year").aggregate(np.sum)
by_year.head()<jupyter_output><empty_output><jupyter_text>Calling `.plot()` on this `DataFrame` results in a more easily interpretable plot:<jupyter_code>by_year.plot();<jupyter_output><empty_output><jupyter_text>So why use `hvplot`? Let us now import `hvplot.pandas` which gives our `DataFrame` objects a new `.hvplot` method which we can call:<jupyter_code>import hvplot.pandas # adds hvplot method to pandas objects

by_year.hvplot()<jupyter_output><empty_output><jupyter_text>Immediately, we can note the following differences from `.plot`:* The plot is not rendered with Matplotlib but with [Bokeh](https://bokeh.org/) instead.* The plot is now interactive: by selecting the various tools available on the toolbar (on the right), you can now pan, box zoom, mouse zoom, save, reset and hover the data respectively.* The last tool (hover) in particular gives you a new view on your data, allowing you to see the exact values on the curve without having to read the values off the axes.  Interpreting and composing plotsLooking at the plot above, we note that the incidence of measles used to be higher in the past and has dropped to nearly zero since the year 1980. What caused this change?With a little research, we may learn that in [1963 a measles vaccine became widely available](https://www.cdc.gov/measles/about/history.html) which brought cases down to negligible levels. This is knowledge that is relevant to this plot and is something that would be useful to annotated explicitly on top of the plot.In this section, we will see how this is easy to achieve using the [HoloViews](http://holoviews.org/) objects returned by `.hvplot`:<jupyter_code>hvplot_obj = by_year.hvplot()
hvplot_obj<jupyter_output><empty_output><jupyter_text>We can look at the textual representation of `hvplot_obj` by printing it:<jupyter_code>print(hvplot_obj)<jupyter_output><empty_output><jupyter_text>Now we see that this is a HoloViews [`Curve`](http://holoviews.org/reference/elements/bokeh/Curve.html) object described in the reference guide [here](http://holoviews.org/reference/elements/bokeh/Curve.html). This object (like all HoloViews objects) is *not a plot* but an object that contains your data, with a rich visual representation. We can see this by looking at the `.data` attribute:<jupyter_code>hvplot_obj.data<jupyter_output><empty_output><jupyter_text>This object can also compose with other HoloViews objects to build rich visualizations. We can see this by importing HoloViews and creating a [`VLine`](http://holoviews.org/reference/elements/bokeh/VLine.html) and a [`Text`](http://holoviews.org/reference/elements/bokeh/Text.html) object:<jupyter_code>import holoviews as hv

vline = hv.VLine(1963).opts(color='black')
text = hv.Text(1963, 27000, " Vaccine introduced", halign='left')<jupyter_output><empty_output><jupyter_text>You can now overlay these on top of the original `hvplot_obj` using the `*` operator, creating a HoloViews [`Overlay`](http://holoviews.org/reference/containers/bokeh/Overlay.html):<jupyter_code>composite = hvplot_obj * vline * text
composite<jupyter_output><empty_output><jupyter_text>We now have an interactive (pannable, zoomable, hoverable) plot with annotations!To inspect the textual representation of this composite object, we can print it:<jupyter_code>print(composite)<jupyter_output><empty_output><jupyter_text>We can now say that we are viewing an overlay consisting of a `Curve`, a `VLine` and some `Text`.  Interactivity with widgetsThe interactive tools offered by [Bokeh](https://bokeh.org/) are one compelling reason to use `hvplot`, but the `hvplot` method offers additional levels of interactivity by generating widgets.To illustrate, let's make a new `DataFrame` that aggregates the measle incidence by `'Year'` while preserving the breakdown by `'State'`:<jupyter_code>measles_agg = df.groupby(['Year', 'State'])['measles'].sum()
measles_agg<jupyter_output><empty_output><jupyter_text>Now we can call `hvplot` to generate a plot over `'Year'` while grouping by `'State'`. The specification of a column to group by results in a dropdown widget by state:<jupyter_code>by_state = measles_agg.hvplot('Year', groupby='State', width=600)
by_state * vline<jupyter_output><empty_output><jupyter_text>Note that you can now view the data for each state while retaining the ability to pan, zoom and hover the plot. In addition, we have the `VLine` marking the point at which the 1963 measles vaccine was introduced.  Plotting large data with `hvplot`So far, the examples have shown how `.hvplot` differs from `.plot` for regular Pandas `DataFrames`. In this example, we will see how `.hvplot` can be used to visualize large volumes of data in [Dask](https://dask.org/) dataframes.First let's import the `airline_flights` sample data from `hvplot`, convert it to a Dask `DataFrame` and view the `.tail` of it:<jupyter_code>from hvplot.sample_data import airline_flights
flights = airline_flights.to_dask().persist()
flights.tail()<jupyter_output><empty_output><jupyter_text>Note that there are 918204 rows in this dataframe! Plotting all these entries (e.g as a scatter plot) is likely to be a slow and memory intensive operation that may well crash the browser tab. We will now see how `hvplot` is able to quickly and efficiently plot all this data regardless.Now we need to import `hvplot.dask` to give our Dask array a `.hvplot` method:<jupyter_code>import hvplot.dask<jupyter_output><empty_output><jupyter_text>We can call the `hvplot.scatter` method to generate a HoloViews [`Scatter`](http://holoviews.org/reference/elements/bokeh/Scatter.html) object (which we will not display the normal way due to the size of the data!)<jupyter_code>scatter = flights.hvplot.scatter(x='distance', y='airtime')
print(scatter)<jupyter_output><empty_output><jupyter_text>Displaying this object normally is risky as it would involve plotting 918204 with Bokeh. However, we can quickly and safely plot it by adding the `datashade=True` keyword:<jupyter_code>flights.hvplot.scatter(x='distance', y='airtime', datashade=True)<jupyter_output><empty_output><jupyter_text>Note that this example is still interactively pannable and zoomable!This is possible due to the use of [datashader](https://datashader.org/) which is a fast rasterizer: our `Scatter` object is rapidly rendered by datshader to an image that is sent to the browser by the Python process. This minimizes the load on the browser by only pushing the image data to the client instead of all 918204 points.  Plotting raster data with `hvplot`So far, all the plots in this notebook have been generated from `DataFrame` style objects. This section shows that `hvplot` can be used to visualize raster data, specifically large [Xarray](http://xarray.pydata.org) datasets.First we import `xarray` and enable `hvplot` support by importing `hvplot.xarray`:<jupyter_code>import xarray as xr
import hvplot.xarray<jupyter_output><empty_output><jupyter_text>Next we load one of the large sample dataset that shops with `xarray`:<jupyter_code>air_ds = xr.tutorial.open_dataset('air_temperature').load()
air = air_ds.air
'Air temperature data has {dims} as dimensions and a shape of {shape}'.format(dims=air.dims, shape=air.shape)<jupyter_output><empty_output><jupyter_text>We can now call `hvplot.scatter` to plot this entire dataset over time, remembering to set `datashade=True`:<jupyter_code>temp_scatter = air.hvplot.scatter('time', groupby=[], datashade=True)
temp_scatter<jupyter_output><empty_output><jupyter_text>Next we can use the `.mean` method on our xarray `DataArray` to average the data over latitude and longitude before plotting it over time with `hvplot.line`:<jupyter_code>temp_mean = air.mean(['lat', 'lon']).hvplot.line('time', color='indianred')
temp_mean<jupyter_output><empty_output><jupyter_text>Finally we can use `*` to easily overlay these two plots:<jupyter_code>temp_scatter * temp_mean<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09326424870466321, 'fraction_numerical': 0.006291635825314582, 'mean_word_length': 6.191616766467066, 'pattern_counts': {'":': 0, '<': 93, '<?xml version=': 0, '>': 93, 'https://': 15, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '23', 'dir': 'jupyter-structured-clean-dedup', 'id': '309076', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3399, 'n_tokens_neox': 3250, 'n_words': 1222, 'path': 'Exploring_Data.ipynb', 'repo_name': 'jlstevens/exploring-data', 'size': '10808'}