<commit_before>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import xarray as xr\n",
    "import holoviews as hv\n",
    "import geoviews as gv\n",
    "import geoviews.feature as gf\n",
    "from cartopy import crs as ccrs\n",
    "\n",
    "hv.extension('matplotlib', 'bokeh')\n",
    "%output size=200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" role=\"alert\">\n",
    "  <strong>Alert:</strong> Some widgets in this notebook are set to render dynamically and require a live notebook server. If you're viewing this notebook on a static website, the widgets won't respond.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main strength of [HoloViews](http://holoviews.org) and its extensions (like GeoViews) is the ability to quickly explore complex datasets by declaring lower-dimensional views into a higher-dimensional space. In HoloViews we refer to the interface that allows you to do this as the conversion API. To begin with we will load a multi-dimensional dataset of surface temperatures for different \"realizations\" (modelling parameters) using [XArray](https://github.com/pydata/xarray/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xr_ensembles = xr.open_dataset('./sample-data/ensembles.nc')\n",
    "xr_ensembles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw in the [Gridded Datasets I Tutorial](Gridded_Datasets_I.ipynb) we can easily wrap this xarray data structure as a HoloViews Dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kdims = ['realization', 'longitude', 'latitude', 'time']\n",
    "vdims = ['surface_temperature']\n",
    "dataset = gv.Dataset(xr_ensembles, kdims=kdims, vdims=vdims, crs=ccrs.PlateCarree())\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the repr we can immediately see the list of key dimensions (time, realization, longitude and latitude) and the value dimension of the cube (surface_temperature). However, unlike most other HoloViews Elements, the ``Dataset`` Element does not display itself visually. This is because it can be n-dimensional and therefore does not have any specific straightforward visual representation on a 2D display. To view the cube, we first have to transform it into individually visualizable chunks.  Before doing so, we will want to supply a custom value formatter for the time dimension so that it is readable by humans:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hv.Dimension.type_formatters[np.datetime64] = '%Y-%m-%d'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conversions\n",
    "\n",
    "A HoloViews Dataset is a wrapper around a complex multi-dimensional datastructure which allows the user to convert their data into individually visualizable views, each usually of lower dimensionality. This is done by grouping the data by some dimension and then casting it to a specific Element type, which visualizes itself.\n",
    "\n",
    "The ``dataset.to`` interface makes this especially easy.  To use it, you supply the Element type that you want to view the data as and the key dimensions of that view and it will figure out the rest. Depending on the type of Element, you can specify one or more dimensions to be displayed. GeoViews provides a set of GeoElements that allow you to display geographic data on a cartographic projection, but you can use any Elements from HoloViews for non-geographic plots.\n",
    "\n",
    "Recall that the cube we are working with has 4 coordinate dimensions (or key dimensions as they are known in HoloViews) -- time, realization, longitude, and latitude.  For our purposes, a geographic plot is defined as a plot that has longitude along the x axis and latitude along the y axis. To declare a two-dimensional geographic plot, we therefore simply request a `gv.Image` plot with ``longitude`` and ``latitude`` as key dimensions.  There is one value dimension (vdim) available, ``surface_temperature``, and any remaining key dimensions (``time`` and ``realization`` in this case) are assigned to a ``HoloMap`` data structure by default.  The resulting ``HoloMap`` gives you widgets automatically to allow you to explore the data across the two \"remaining\" key dimensions (those not mapped onto axes of the image):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "geo_dims = ['longitude', 'latitude']\n",
    "(dataset.to(gv.Image, geo_dims) * gf.coastline)[::5, ::5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this way we can visualize the geographic data in a number of ways, currently either as an ``Image`` (as above) or as ``LineContours``, ``FilledContours``, or ``Points``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%opts Points [color_index=2 size_index=None] (cmap='jet')\n",
    "hv.Layout([dataset.to(el, geo_dims)[::10, ::10] * gf.coastline\n",
    "           for el in [gv.FilledContours, gv.LineContours, gv.Points]]).cols(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that by default the conversion interface will automatically expand all the individual Elements, which can take some time if the data is very large. Instead we can also request the objects to be expanded dynamically using the ``dynamic`` keyword:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.to(gv.Image, geo_dims, dynamic=True) * gf.coastline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using ``dynamic`` mode means that the data for each frame is only extracted when you're actually viewing that part of the data, which can have huge benefits in terms of speed and memory consumption.  However, it relies on having a running Python process to render and serve each image, and so it cannot be used when generating static HTML output such as for the GeoViews web site.\n",
    "\n",
    "\n",
    "## Non-geographic views\n",
    "\n",
    "So far we have focused entirely on geographic views of the data, plotting the data on a projection. However the conversion interface is completely general, allowing us to slice and dice the data in any way we like. For these views we will switch to the bokeh plotting extension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [<commit_msg>Updated docs with QuadMesh example<commit_after>{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import xarray as xr\n",
    "import holoviews as hv\n",
    "import geoviews as gv\n",
    "import geoviews.feature as gf\n",
    "from cartopy import crs as ccrs\n",
    "\n",
    "hv.extension('matplotlib', 'bokeh')\n",
    "%output size=200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\" role=\"alert\">\n",
    "  <strong>Alert:</strong> Some widgets in this notebook are set to render dynamically and require a live notebook server. If you're viewing this notebook on a static website, the widgets won't respond.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main strength of [HoloViews](http://holoviews.org) and its extensions (like GeoViews) is the ability to quickly explore complex datasets by declaring lower-dimensional views into a higher-dimensional space. In HoloViews we refer to the interface that allows you to do this as the conversion API. To begin with we will load a multi-dimensional dataset of surface temperatures for different \"realizations\" (modelling parameters) using [XArray](https://github.com/pydata/xarray/):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xr_ensembles = xr.open_dataset('./sample-data/ensembles.nc')\n",
    "xr_ensembles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw in the [Gridded Datasets I Tutorial](Gridded_Datasets_I.ipynb) we can easily wrap this xarray data structure as a HoloViews Dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = gv.Dataset(xr_ensembles, vdims='surface_temperature', crs=ccrs.PlateCarree())\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the repr we can immediately see the list of key dimensions (time, realization, longitude and latitude) and the value dimension of the cube (surface_temperature). However, unlike most other HoloViews Elements, the ``Dataset`` Element does not display itself visually. This is because it can be n-dimensional and therefore does not have any specific straightforward visual representation on a 2D display. To view the cube, we first have to transform it into individually visualizable chunks.  Before doing so, we will want to supply a custom value formatter for the time dimension so that it is readable by humans:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hv.Dimension.type_formatters[np.datetime64] = '%Y-%m-%d'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conversions\n",
    "\n",
    "A HoloViews Dataset is a wrapper around a complex multi-dimensional datastructure which allows the user to convert their data into individually visualizable views, each usually of lower dimensionality. This is done by grouping the data by some dimension and then casting it to a specific Element type, which visualizes itself.\n",
    "\n",
    "The ``dataset.to`` interface makes this especially easy.  To use it, you supply the Element type that you want to view the data as and the key dimensions of that view and it will figure out the rest. Depending on the type of Element, you can specify one or more dimensions to be displayed. GeoViews provides a set of GeoElements that allow you to display geographic data on a cartographic projection, but you can use any Elements from HoloViews for non-geographic plots.\n",
    "\n",
    "Recall that the cube we are working with has 4 coordinate dimensions (or key dimensions as they are known in HoloViews) -- time, realization, longitude, and latitude.  For our purposes, a geographic plot is defined as a plot that has longitude along the x axis and latitude along the y axis. To declare a two-dimensional geographic plot, we therefore simply request a `gv.Image` plot with ``longitude`` and ``latitude`` as key dimensions.  There is one value dimension (vdim) available, ``surface_temperature``, and any remaining key dimensions (``time`` and ``realization`` in this case) are assigned to a ``HoloMap`` data structure by default.  The resulting ``HoloMap`` gives you widgets automatically to allow you to explore the data across the two \"remaining\" key dimensions (those not mapped onto axes of the image):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "geo_dims = ['longitude', 'latitude']\n",
    "(dataset.to(gv.Image, geo_dims) * gf.coastline)[::5, ::5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this way we can visualize the geographic data in a number of ways, currently either as an ``Image`` (as above) or as ``LineContours``, ``FilledContours``, or ``Points``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%opts Points [color_index=2 size_index=None] (cmap='jet')\n",
    "hv.Layout([dataset.to(el, geo_dims)[::10, ::10] * gf.coastline\n",
    "           for el in [gv.FilledContours, gv.LineContours, gv.Points]]).cols(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that by default the conversion interface will automatically expand all the individual Elements, which can take some time if the data is very large. Instead we can also request the objects to be expanded dynamically using the ``dynamic`` keyword:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.to(gv.Image, geo_dims, dynamic=True) * gf.coastline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using ``dynamic`` mode means that the data for each frame is only extracted when you're actually viewing that part of the data, which can have huge benefits in terms of speed and memory consumption.  However, it relies on having a running Python process to render and serve each image, and so it cannot be used when generating static HTML output such as for the GeoViews web site.\n",
    "\n",
    "## Irregularly sampled data\n",
    "\n",
    "Often gridded datasets are not regularly sampled, instead providing irregularly sampled multi-dimensional coordinates. Such datasets can be easily visualized using the QuadMesh element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%opts QuadMesh [colorbar=True fig_size=300 projection=ccrs.Robinson()] (cmap='RdBu_r')\n",
    "xrds = xr.tutorial.load_dataset('RASM_example_data')\n",
    "qmesh = gv.Dataset(xrds.Tair).to(gv.QuadMesh, ['xc', 'yc'], dynamic=True)\n",
    "qmesh.redim.range(Tair=(-30, 30)) * gf.coastline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Non-geographic views\n",
    "\n",
    "So far we have focused entirely on geographic views of the data, plotting the data on a projection. However the conversion interface is completely general, allowing us to slice and dice the data in any way we like. For these views we will switch to the bokeh plotting extension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
{'fraction_non_alphanumeric': 0.12849313214696528, 'fraction_numerical': 0.002503552337776575, 'mean_word_length': 3.6000622471210706, 'pattern_counts': {'":': 140, '<': 11, '<?xml version=': 0, '>': 11, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4201348', 'n_tokens_mistral': 4545, 'n_tokens_neox': 4406, 'n_words': 1713}