<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href='http://www.holoviews.org'><img src=\"assets/hv+bk.png\" alt=\"HV+BK logos\" width=\"40%;\" align=\"left\"/></a>\n",
    "<div style=\"float:right;\"><h2>07. Working with large datasets</h2></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HoloViews supports even high-dimensional datasets easily, and the standard mechanisms discussed already work well as long as you select a small enough subset of the data to display at any one time.  However, some datasets are just inherently large, even for a single frame of data, and cannot safely be transferred for display in any standard web browser. Luckily, HoloViews makes it simple for you to use the separate [``datashader``](http://datashader.readthedocs.io/en/latest/) together with any of the plotting extension libraries, including Bokeh and Matplotlib. The datashader library is designed to complement standard plotting libraries by providing faithful visualizations for very large datasets, focusing on revealing the overall distribution, not just individual data points.\n",
    "\n",
    "Datashader uses computations accellerated using [Numba](http://numba.pydata.org), making it fast to work with datasets of millions or billions of datapoints stored in [``dask``](http://dask.pydata.org/en/latest/) dataframes.  Dask dataframes provide an API that is functionally equivalent to pandas, but allows working with data out of core while scaling out to many processors and even clusters. Here we will use Dask to load a large CSV file of taxi coordinates.\n",
    "\n",
    "<div >\n",
    "<img align=\"left\" src=\"./assets/numba.png\" width='140px'></img>\n",
    "<img align=\"left\" src=\"./assets/dask.svg\" width='115px'></img>\n",
    "<img align=\"left\" src=\"./assets/datashader.png\" width='158px'></img>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does datashader work?\n",
    "\n",
    "<img src=\"./assets/datashader_pipeline.png\" width=\"80%\"></img>\n",
    "\n",
    "* Tools like Bokeh map **Data** (left) directly into an HTML/JavaScript **Plot** (right)\n",
    "* datashader instead renders **Data** into a plot-sized **Aggregate** array, from which an **Image** can be constructed then embedded into a Bokeh **Plot**\n",
    "* Only the fixed-sized **Image** needs to be sent to the browser, allowing millions or billions of datapoints to be used\n",
    "* Every step automatically adjusts to the data, but can be customized"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When not to use datashader\n",
    "\n",
    "* Plotting less than 1e5 or 1e6 data points\n",
    "* When every datapoint matters; standard Bokeh will render all of them\n",
    "* For full interactivity (hover tools) with every datapoint\n",
    "\n",
    "#### When to use datashader\n",
    "\n",
    "* Actual big data; when Bokeh/Matplotlib have trouble\n",
    "* When the distribution matters more than individual points\n",
    "* When you find yourself sampling or binning to better understand the distribution"<commit_msg>Updated solutions notebook to fix GitHub rendering<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href='http://www.holoviews.org'><img src=\"assets/hv+bk.png\" alt=\"HV+BK logos\" width=\"40%;\" align=\"left\"/></a>\n",
    "<div style=\"float:right;\"><h2>07. Working with large datasets</h2></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HoloViews supports even high-dimensional datasets easily, and the standard mechanisms discussed already work well as long as you select a small enough subset of the data to display at any one time.  However, some datasets are just inherently large, even for a single frame of data, and cannot safely be transferred for display in any standard web browser. Luckily, HoloViews makes it simple for you to use the separate [``datashader``](http://datashader.readthedocs.io/en/latest/) together with any of the plotting extension libraries, including Bokeh and Matplotlib. The datashader library is designed to complement standard plotting libraries by providing faithful visualizations for very large datasets, focusing on revealing the overall distribution, not just individual data points.\n",
    "\n",
    "Datashader uses computations accellerated using [Numba](http://numba.pydata.org), making it fast to work with datasets of millions or billions of datapoints stored in [``dask``](http://dask.pydata.org/en/latest/) dataframes.  Dask dataframes provide an API that is functionally equivalent to pandas, but allows working with data out of core while scaling out to many processors and even clusters. Here we will use Dask to load a large CSV file of taxi coordinates.\n",
    "\n",
    "<div>\n",
    "<img align=\"left\" src=\"./assets/numba.png\" width='140px'/>\n",
    "<img align=\"left\" src=\"./assets/dask.svg\" width='115px'/>\n",
    "<img align=\"left\" src=\"./assets/datashader.png\" width='158px'/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does datashader work?\n",
    "\n",
    "<img src=\"./assets/datashader_pipeline.png\" width=\"80%\"/>\n",
    "\n",
    "* Tools like Bokeh map **Data** (left) directly into an HTML/JavaScript **Plot** (right)\n",
    "* datashader instead renders **Data** into a plot-sized **Aggregate** array, from which an **Image** can be constructed then embedded into a Bokeh **Plot**\n",
    "* Only the fixed-sized **Image** needs to be sent to the browser, allowing millions or billions of datapoints to be used\n",
    "* Every step automatically adjusts to the data, but can be customized"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### When not to use datashader\n",
    "\n",
    "* Plotting less than 1e5 or 1e6 data points\n",
    "* When every datapoint matters; standard Bokeh will render all of them\n",
    "* For full interactivity (hover tools) with every datapoint\n",
    "\n",
    "#### When to use datashader\n",
    "\n",
    "* Actual big data; when Bokeh/Matplotlib have trouble\n",
    "* When the distribution matters more than individual points\n",
    "* When you find yourself sampling or binning to better understand the distribution"
{'fraction_non_alphanumeric': 0.14431082331174838, 'fraction_numerical': 0.0064754856614246065, 'mean_word_length': 3.6736311239193085, 'pattern_counts': {'":': 26, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3631709', 'n_tokens_mistral': 1945, 'n_tokens_neox': 1909, 'n_words': 734}