<commit_before>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AGU 2018 Demo\n",
    "\n",
    "This notebook shows how to use `birdy`'s high-level interface to WPS processes. \n",
    "\n",
    "Here we access a test server called `Emu` offering a dozen or so dummy processes. \n",
    "\n",
    "## The shell interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: birdy [OPTIONS] COMMAND [ARGS]...\n",
      "\n",
      "  Birdy is a command line client for Web Processing Services.\n",
      "\n",
      "  Documentation is available on readthedocs:\n",
      "  http://birdy.readthedocs.org/en/latest/\n",
      "\n",
      "Options:\n",
      "  --version         Show the version and exit.\n",
      "  --cert TEXT       Client side certificate containing both certificate and\n",
      "                    private key.\n",
      "  -S, --send        Send client side certificate to WPS. Default: false\n",
      "  -s, --sync        Execute process in sync mode. Default: async mode.\n",
      "  -t, --token TEXT  Token to access the WPS service.\n",
      "  -h, --help        Show this message and exit.\n",
      "\n",
      "Commands:\n",
      "  ultimate_question         Answer to the ultimate question: This process...\n",
      "  sleep                     Sleep Process: Testing a long running...\n",
      "  nap                       Afternoon Nap (supports sync calls only):...\n",
      "  bbox                      Bounding box in- and out: Give bounding box,...\n",
      "  hello                     Say Hello: Just says a friendly Hello.Returns...\n",
      "  dummyprocess              Dummy Process: DummyProcess to check the WPS...\n",
      "  wordcounter               Word Counter: Counts words in a given text.\n",
      "  chomsky                   Chomsky text generator: Generates a random...\n",
      "  inout                     In and Out: Testing all WPS input and output...\n",
      "  binaryoperatorfornumbers  Binary Operator for Numbers: Performs...\n",
      "  show_error                Show a WPS Error: This process will fail...\n",
      "  multiple_outputs          Multiple Outputs: Produces multiple files and...\n",
      "  esgf_demo                 ESGF Demo: Shows how to use WPS metadata for...\n",
      "  output_formats            Return different output formats.: Dummy...\n",
      "  poly_centroid             Approximate centroid of a polygon.: Return...\n",
      "  ncmeta                    Show NetCDF Metadata: Dataset can be either a...\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export WPS_SERVICE=https://bovec.dkrz.de/ows/proxy/emu?Service=WPS&Request=GetCapabilities&Version=1.0.0\n",
    "birdy -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: birdy hello [OPTIONS]\n",
      "\n",
      "  Say Hello: Just says a friendly Hello.Returns a literal string output with\n",
      "  Hello plus the inputed name.\n",
      "\n",
      "Options:\n",
      "  --version    Show the version and exit.\n",
      "  --name TEXT  Your name\n",
      "  -h, --help   Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "birdy hello -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProcessSucceeded\n",
      "Output:\n",
      "output=['Hello stranger']\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "birdy hello --name stranger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The python interface\n",
    "\n",
    "The `WPSClient` function creates a *mock* python module whose functions actually call a remote WPS process. The \n",
    "docstring and signature of the function are dynamically created from the remote's process description. If you type `wps.` and then press `Tab`, you should see a drop-down list of available processes. Simply call `help` on each process of type `?` after the process to print the docstring for that process. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on method binaryoperatorfornumbers in module birdy.client.base:\n",
      "\n",
      "binaryoperatorfornumbers(inputa=2.0, inputb=3.0, operator='add') method of birdy.client.base.WPSClient instance\n",
      "    Performs operation on two numbers and returns the answer. This example process is taken from Climate4Impact.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    inputa : //www.w3.org/TR/xmlschema-2/#float\n",
      "        Enter Input 1\n",
      "    inputb : //www.w3.org/TR/xmlschema-2/#float\n",
      "        Enter Input 2\n",
      "    operator : {'add', 'substract', 'divide', 'multiply'}//www.w3.org/TR/xmlschema-2/#string\n",
      "        Choose a binary Operator\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    output : //www.w3.org/TR/xmlschema-2/#float\n",
      "        Binary operator result\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from birdy import WPSClient\n",
    "url = \"https://bovec.dkrz.de/ows/proxy/emu?Service=WPS&Request=GetCapabilities&Version=1.0.0\"\n",
    "wps = WPSClient(url)\n",
    "help(wps.binaryoperatorfornumbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process execution\n",
    "\n",
    "Processes are executed by calling the function. Each process instantaneoulsy returns a `WPSExecute` object. The actual output values of the process are obtained by calling the `get` method. This `get` method returns a namedtuple storing the process outputs as native python objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<birdy.client.utils.WPSExecution object at 0x7f2b844716a0>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "binaryoperatorfornumbersResponse(\n",
       "    output=3.0\n",
       ")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp = wps.binaryoperatorfornumbers(1, 2, operator='add')\n",
    "print(resp)\n",
    "resp.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, the `inout` function returns a wide variety of data types (float, integers, dates, etc) all of which are converted into a corresponding python type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "inoutResponse(\n",
       "    string='This is just a string',\n",
       "    int=7,\n",
       "    float=3.14,\n",
       "    boolean=True,\n",
       "    angle=90.0,\n",
       "    time=datetime.time(12, 0),\n",
       "    date=datetime.date(2012, 5, 1),\n",
       "    datetime=datetime.datetime(2016, 9, 2, 12, 0, tzinfo=tzutc()),\n",
       "    string_choice='scissor',\n",
       "    string_multiple_choice='gentle albatros',\n",
       "    text=ComplexDataInput(mimeType=None, encoding=None, schema=None, value='request didn&#39;t have a text file.'),\n",
       "    dataset=ComplexDataInput(mimeType=None, encoding=None, schema=None, value='request didn&#39;t have a netcdf file.'),\n",
       "    bbox=None\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = wps.inout().get()\n",
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving outputs by references\n",
    "\n",
    "For `ComplexData` objects, WPS servers often return a reference to the output (an http link) instead of the actual data. This is useful if that output is to serve as an input to another process, so as to avoid passing back and forth large files for nothing. \n",
    "\n",
    "With `birdy`, the outputs are by default return values are the references themselves, but it's also possible to download these references in the background and convert them into python objects. To trigger this automatic conversion, set `asobj` to `True` when calling the `get` method. In the example below, we're using a dummy process called `output_formats`, whose first output is a netCDF file, and second output is a json file. With `asobj=True`, the netCDF file is opened and returned as a `netcdf4.Dataset` instance, and the json file into a dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output_formatsResponse(\n",
      "    netcdf='https://bovec.dkrz.de/download/outputs/emu/2338dbb0-fc9a-11e8-8da0-109836a7cf3a/dummy.nc',\n",
      "    json='https://bovec.dkrz.de/download/outputs/emu/2338dbb0-fc9a-11e8-8da0-109836a7cf3a/dummy.json'\n",
      ")\n",
      "{'testing': [1, 2]}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<class 'netCDF4._netCDF4.Dataset'>\n",
       "root group (NETCDF4 data model, file format HDF5):\n",
       "    title: Test dataset\n",
       "    dimensions(sizes): time(1)\n",
       "    variables(dimensions): float64 \u001b[4mtime\u001b[0m(time)\n",
       "    groups: "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = wps.output_formats()\n",
    "nc, json = out.get()\n",
    "print(out.get())\n",
    "ds, json = out.get(asobj=True)\n",
    "print(json)\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Progress bar\n",
    "\n",
    "It's possible to display a progress bar when calling a process. The interface to do so at the moment goes like this. Note that the cancel button does not do much here, as the WPS server does not support interruption requests. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f13f91b55a946a7bb3a3ba88d80bf58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, bar_style='info', description='Processing:'), Button(button_style='danger'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wps_i = WPSClient(url, progress=True)\n",
    "out = wps_i.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
<commit_msg>brought back agu notebook from the dead<commit_after>{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AGU 2018 Demo\n",
    "\n",
    "This notebook shows how to use `birdy`'s high-level interface to WPS processes. \n",
    "\n",
    "Here we access a test server called `Emu` offering a dozen or so dummy processes. \n",
    "\n",
    "## The shell interface"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: birdy [OPTIONS] COMMAND [ARGS]...\n",
      "\n",
      "  Birdy is a command line client for Web Processing Services.\n",
      "\n",
      "  Documentation is available on readthedocs:\n",
      "  http://birdy.readthedocs.org/en/latest/\n",
      "\n",
      "Options:\n",
      "  --version         Show the version and exit.\n",
      "  --cert TEXT       Client side certificate containing both certificate and\n",
      "                    private key.\n",
      "  -S, --send        Send client side certificate to WPS. Default: false\n",
      "  -s, --sync        Execute process in sync mode. Default: async mode.\n",
      "  -t, --token TEXT  Token to access the WPS service.\n",
      "  -h, --help        Show this message and exit.\n",
      "\n",
      "Commands:\n",
      "  ultimate_question         Answer to the ultimate question: This process...\n",
      "  sleep                     Sleep Process: Testing a long running...\n",
      "  nap                       Afternoon Nap (supports sync calls only):...\n",
      "  bbox                      Bounding box in- and out: Give bounding box,...\n",
      "  hello                     Say Hello: Just says a friendly Hello.Returns...\n",
      "  dummyprocess              Dummy Process: DummyProcess to check the WPS...\n",
      "  wordcounter               Word Counter: Counts words in a given text.\n",
      "  chomsky                   Chomsky text generator: Generates a random...\n",
      "  inout                     In and Out: Testing all WPS input and output...\n",
      "  binaryoperatorfornumbers  Binary Operator for Numbers: Performs...\n",
      "  show_error                Show a WPS Error: This process will fail...\n",
      "  multiple_outputs          Multiple Outputs: Produces multiple files and...\n",
      "  esgf_demo                 ESGF Demo: Shows how to use WPS metadata for...\n",
      "  output_formats            Return different output formats.: Dummy...\n",
      "  poly_centroid             Approximate centroid of a polygon.: Return...\n",
      "  ncmeta                    Return NetCDF Metadata: Return metadata from...\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "export WPS_SERVICE=https://bovec.dkrz.de/ows/proxy/emu?Service=WPS&Request=GetCapabilities&Version=1.0.0\n",
    "birdy -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: birdy hello [OPTIONS]\n",
      "\n",
      "  Say Hello: Just says a friendly Hello.Returns a literal string output with\n",
      "  Hello plus the inputed name.\n",
      "\n",
      "Options:\n",
      "  --version    Show the version and exit.\n",
      "  --name TEXT  Your name\n",
      "  -h, --help   Show this message and exit.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "birdy hello -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProcessSucceeded\n",
      "Output:\n",
      "output=['Hello stranger']\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "birdy hello --name stranger"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The python interface\n",
    "\n",
    "The `WPSClient` function creates a *mock* python module whose functions actually call a remote WPS process. The \n",
    "docstring and signature of the function are dynamically created from the remote's process description. If you type `wps.` and then press `Tab`, you should see a drop-down list of available processes. Simply call `help` on each process of type `?` after the process to print the docstring for that process. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on method binaryoperatorfornumbers in module birdy.client.base:\n",
      "\n",
      "binaryoperatorfornumbers(inputa=2.0, inputb=3.0, operator='add') method of birdy.client.base.WPSClient instance\n",
      "    Performs operation on two numbers and returns the answer. This example process is taken from Climate4Impact.\n",
      "    \n",
      "    Parameters\n",
      "    ----------\n",
      "    inputa : //www.w3.org/TR/xmlschema-2/#float\n",
      "        Enter Input 1\n",
      "    inputb : //www.w3.org/TR/xmlschema-2/#float\n",
      "        Enter Input 2\n",
      "    operator : {'add', 'substract', 'divide', 'multiply'}//www.w3.org/TR/xmlschema-2/#string\n",
      "        Choose a binary Operator\n",
      "    \n",
      "    Returns\n",
      "    -------\n",
      "    output : //www.w3.org/TR/xmlschema-2/#float\n",
      "        Binary operator result\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from birdy import WPSClient\n",
    "url = \"https://bovec.dkrz.de/ows/proxy/emu?Service=WPS&Request=GetCapabilities&Version=1.0.0\"\n",
    "wps = WPSClient(url)\n",
    "help(wps.binaryoperatorfornumbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Process execution\n",
    "\n",
    "Processes are executed by calling the function. Each process instantaneoulsy returns a `WPSExecute` object. The actual output values of the process are obtained by calling the `get` method. This `get` method returns a namedtuple storing the process outputs as native python objects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<birdy.client.utils.WPSExecution object at 0x7f69dc9860b8>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "binaryoperatorfornumbersResponse(\n",
       "    output=3.0\n",
       ")"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp = wps.binaryoperatorfornumbers(1, 2, operator='add')\n",
    "print(resp)\n",
    "resp.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, the `inout` function returns a wide variety of data types (float, integers, dates, etc) all of which are converted into a corresponding python type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "inoutResponse(\n",
       "    string='This is just a string',\n",
       "    int=7,\n",
       "    float=3.14,\n",
       "    boolean=True,\n",
       "    angle=90.0,\n",
       "    time=datetime.time(12, 0),\n",
       "    date=datetime.date(2012, 5, 1),\n",
       "    datetime=datetime.datetime(2016, 9, 2, 12, 0, tzinfo=tzutc()),\n",
       "    string_choice='scissor',\n",
       "    string_multiple_choice='gentle albatros',\n",
       "    text=ComplexDataInput(mimeType=None, encoding=None, schema=None, value='request didn&#39;t have a text file.'),\n",
       "    dataset=ComplexDataInput(mimeType=None, encoding=None, schema=None, value='request didn&#39;t have a netcdf file.'),\n",
       "    bbox=None\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wps.inout().get()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving outputs by references\n",
    "\n",
    "For `ComplexData` objects, WPS servers often return a reference to the output (an http link) instead of the actual data. This is useful if that output is to serve as an input to another process, so as to avoid passing back and forth large files for nothing. \n",
    "\n",
    "With `birdy`, the outputs are by default return values are the references themselves, but it's also possible to download these references in the background and convert them into python objects. To trigger this automatic conversion, set `asobj` to `True` when calling the `get` method. In the example below, we're using a dummy process called `output_formats`, whose first output is a netCDF file, and second output is a json file. With `asobj=True`, the netCDF file is opened and returned as a `netcdf4.Dataset` instance, and the json file into a dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output_formatsResponse(\n",
      "    netcdf='https://bovec.dkrz.de/download/outputs/emu/16ceb4ce-fda0-11e8-8da0-109836a7cf3a/dummy.nc',\n",
      "    json='https://bovec.dkrz.de/download/outputs/emu/16ceb4ce-fda0-11e8-8da0-109836a7cf3a/dummy.json'\n",
      ")\n",
      "{'testing': [1, 2]}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<class 'netCDF4._netCDF4.Dataset'>\n",
       "root group (NETCDF4 data model, file format HDF5):\n",
       "    title: Test dataset\n",
       "    dimensions(sizes): time(1)\n",
       "    variables(dimensions): float64 \u001b[4mtime\u001b[0m(time)\n",
       "    groups: "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = wps.output_formats()\n",
    "nc, json = out.get()\n",
    "print(out.get())\n",
    "ds, json = out.get(asobj=True)\n",
    "print(json)\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Progress bar\n",
    "\n",
    "It's possible to display a progress bar when calling a process. The interface to do so at the moment goes like this. Note that the cancel button does not do much here, as the WPS server does not support interruption requests. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "07f83f49ea69467693280db1aaf16195",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(IntProgress(value=0, bar_style='info', description='Processing:'), Button(button_style='danger'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wps_i = WPSClient(url, progress=True)\n",
    "out = wps_i.sleep(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

{'fraction_non_alphanumeric': 0.16339953572349755, 'fraction_numerical': 0.014315192158885737, 'mean_word_length': 2.2206839263463936, 'pattern_counts': {'":': 244, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 8, 'lorem ipsum': 0, 'www.': 8, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6410678', 'n_tokens_mistral': 8072, 'n_tokens_neox': 7713, 'n_words': 1986}