<commit_before>.. :author: Continuuity, Inc.
   :version: 2.3.0
   :description: HTTP Interface to the Continuuity Reactor

=================================
Continuuity Reactor HTTP REST API
=================================

.. reST Editor: .. section-numbering::
.. reST Editor: .. contents::

.. rst2pdf: .. contents::
.. rst2pdf: config _templates/pdf-config
.. rst2pdf: stylesheets _templates/pdf-stylesheet
.. rst2pdf: build ../build-pdf/

Introduction
============

The Continuuity Reactor has an HTTP interface for a multitude of purposes:

- **Stream:** sending data events to a Stream, or to inspect the contents of a Stream
- **Dataset:** interacting with Datasets, Dataset Modules, and Dataset Types
- **Data:** interacting with Datasets (deprecated)
- **Query:** sending ad-hoc queries to Reactor Datasets
- **Procedure:** sending calls to a stored Procedure
- **Reactor Client:** deploying and managing Applications, and managing the life cycle of Flows,
  Procedures, MapReduce jobs, Workflows, and Custom Services
- **Logging:** retrieving Application logs
- **Metrics:** retrieving metrics for system and user Applications (user-defined metrics)
- **Monitor:** checking the status of various Reactor services, both System and Custom

**Note:** The HTTP interface binds to port ``10000``. This port cannot be changed.

Conventions
-----------

In this API, *client* refers to an external application that is calling the Continuuity Reactor using the HTTP interface.

In this API, *Application* refers to a user Application that has been deployed into the Continuuity Reactor.

All URLs referenced in this API have this base::

	http://<gateway>:10000/v2

where ``<gateway>`` is the URL of the Continuuity Reactor. The base URL is represented as::

	<base-url>

For example::

	PUT <base-url>/streams/<new-stream-id>

means
::

	PUT http://<gateway>:10000/v2/streams/<new-stream-id>
	

Text that are variables that you are to replace is indicated by a series of angle brackets (``< >``). For example::

	PUT <base-url>/streams/<new-stream-id>

indicates that—in addition to the ``<base-url>``—the text ``<new-stream-id>`` is a variable
and that you are to replace it with your value, perhaps in this case *mystream*::

	PUT <base-url>/streams/mystream

.. rst2pdf: PageBreak

Status Codes
------------

`Common status codes <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html>`__ returned for all HTTP calls:


.. list-table::
   :widths: 10 30 60
   :header-rows: 1

   * - Code
     - Description
     - Explanation
   * - ``200``
     - ``OK``
     - The request returned successfully
   * - ``400``
     - ``Bad Request``
     - The request had a combination of parameters that is not recognized
   * - ``401``
     - ``Unauthorized``
     - The request did not contain an authentication token
   * - ``403``
     - ``Forbidden``
     - The request was authenticated but the client does not have permission
   * - ``404``
     - ``Not Found``
     - The request did not address any of the known URIs
   * - ``405``
     - ``Method Not Allowed``
     - A request was received with a method not supported for the URI
   * - ``409``
     - ``Conflict``
     - A request could not be completed due to a conflict with the current resource state
   * - ``500``
     - ``Internal Server Error``
     - An internal error occurred while processing the request
   * - ``501``
     - ``Not Implemented``
     - A request contained a query that is not supported by this API

**Note:** These returned status codes are not necessarily included in the descriptions of the API,
but a request may return any of these.


SSL Required for Sandboxed Continuuity Reactor
----------------------------------------------
When you interact with a Sandboxed Continuuity Reactor,
the Continuuity HTTP APIs require that you use SSL for the connection
and that you authenticate your request by sending your API key in an HTTP header::

	X-Continuuity-ApiKey: <api-key>

.. list-table::
   :widths: 15 85
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<api-key>``
     - Continuuity Reactor API key, obtained from an account at
       `Continuuity Accounts <http://accounts.continuuity.com>`__


Working with Reactor Security
-----------------------------
When working with a Reactor cluster with security enabled (``security.enabled=true`` in
``continuuity-site.xml``), all calls to the HTTP APIs must be authenticated.  Clients must first
obtain an access token from the authentication server (see the "Security" section of the
guide). In order to authenticate, all client requests must supply this access token in the
``Authorization`` header of the request::

   Authorization: Bearer <PASSWORD>

For Reactor-issued access tokens, the authentication scheme must always be ``Bearer``.


Stream HTTP API
===============
This interface supports creating Streams, sending events to a Stream, and reading single events from a Stream.

Streams may have multiple consumers (for example, multiple Flows), each of which may be a group of different agents (for example, multiple instances of a Flowlet).

In order to read events from a Stream, a client application must first obtain a consumer (group) id, which is then passed to subsequent read requests.


Creating a Stream
-----------------
A Stream can be created with an HTTP PUT method to the URL::

	PUT <base-url>/streams/<new-stream-id>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<new-stream-id>``
     - Name of the Stream to be created

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event either successfully created a Stream or the Stream already exists

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/streams/mystream``
   * - Description
     - Create a new Stream named *mystream*

Comments
........
- The ``<new-stream-id>`` should only contain ASCII letters, digits and hyphens.
- If the Stream already exists, no error is returned, and the existing Stream remains in place.

.. rst2pdf: PageBreak

Sending Events to a Stream
--------------------------
An event can be sent to a Stream by sending an HTTP POST method to the URL of the Stream::

	POST <base-url>/streams/<stream-id>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received
   * - ``404 Not Found``
     - The Stream does not exist

:Note: The response will always have an empty body

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream``
   * - Description
     - Send an event to the existing Stream named *mystream*

Comments
........
- The body of the request must contain the event in binary form.
- You can pass headers for the event as HTTP headers by prefixing them with the *stream-id*::

	<stream-id>.<property>:<string value>

  After receiving the request, the HTTP handler transforms it into a Stream event:

  #. The body of the event is an identical copy of the bytes
     found in the body of the HTTP post request.
  #. If the request contains any headers prefixed with the *stream-id*,
     the *stream-id* prefix is stripped from the header name and
     the header is added to the event.

.. rst2pdf: PageBreak

Reading Events from a Stream: Getting a Consumer-ID
---------------------------------------------------
Get a *Consumer-ID* for a Stream by sending an HTTP POST method to the URL::

	POST <base-url>/streams/<stream-id>/consumer-id

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

HTTP Responses
..............
.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and a new ``consumer-id`` was returned
   * - ``404 Not Found``
     - The Stream does not exist

Example
.......
.. list-table::
   :widths: 30 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream/consumer-id``
   * - Description
     - Request a ``Consumer-ID`` for the Stream named *mystream*

Comments
........
- Streams may have multiple consumers (for example, multiple Flows), 
  each of which may be a group of different agents (for example, multiple instances of a Flowlet).
- In order to read events from a Stream, a client application must
  first obtain a consumer (group) id, which is then passed to subsequent read requests.
- The ``Consumer-ID`` is returned in a response header and—for convenience—also in the body of the response::

	X-Continuuity-ConsumerId: <consumer-id>

  Once you have the ``Consumer-ID``, single events can be read from the Stream.

.. rst2pdf: PageBreak

Reading Events from a Stream: Using the Consumer-ID
---------------------------------------------------
A read is performed as an HTTP POST method to the URL::

	POST <base-url>/streams/<stream-id>/dequeue

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

The request must pass the ``Consumer-ID`` in a header of the form::

	X-Continuuity-ConsumerId: <consumer-id>

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the result of the read was returned
   * - ``204 No Content``
     - The Stream exists but it is either empty or the given ``Consumer-ID``
       has read all the events in the Stream
   * - ``404 Not Found``
     - The Stream does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream/dequeue``
   * - Description
     - Read the next event from an existing Stream named *mystream*

Comments
........
The read will always return the next event from the Stream that was inserted first and has not been read yet
(first-in, first-out or FIFO semantics). If the Stream has never been read from before, the first event will be read.

For example, in order to read the third event that was sent to a Stream,
two previous reads have to be performed after receiving the ``Consumer-ID``.
You can always start reading from the first event by getting a new ``Consumer-ID``.

The response will contain the binary body of the event in its body and a header for each header of the Stream event,
analogous to how you send headers when posting an event to the Stream::

	<stream-id>.<property>:<value>

.. rst2pdf: PageBreak

Truncating a Stream
-------------------
Truncation means the deletion of all events that were written to the Stream. 
This is permanent and cannot be undone.
A Stream can be truncated with an HTTP POST method to the URL::

	POST <base-url>/streams/<stream-id>/truncate

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The Stream was successfully truncated
   * - ``404 Not Found``
     - The Stream ``<stream-id>`` does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream/truncate``
   * - Description
     - Delete all events in the Stream named *mystream*

.. rst2pdf: PageBreak

Setting Time-To-Live Property of a Stream
------------------------------------------
The Time-To-Live (TTL) property governs how long an event is valid for consumption since 
it was written to the Stream.
The default TTL for all Streams is infinite, meaning that events will never expire.
The TTL property of a Stream can be changed with an HTTP PUT method to the URL::

	PUT <base-url>/streams/<stream-id>/config

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

The new TTL value is passed in the request body as::

	{ "ttl" : <ttl-in-seconds> }

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<ttl-in-seconds>``
     - Number of seconds that an event will be valid for since ingested

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The stream TTL was changed successfully
   * - ``400 Bad Request``
     - The TTL value is not a non-negative integer
   * - ``404 Not Found``
     - The Stream does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/streams/mystream/config``

       with the new TTL value as a JSON string in the body::

	  { "ttl" : 86400 }
     
   * - Description
     - Change the TTL property of the Stream named *mystream* to 1 day


Reading Multiple Events
-----------------------
Reading multiple events is not supported directly by the Stream HTTP API,
but the command-line tool ``stream-client`` demonstrates how to view *all*, the *first N*, or the *last N* events in the Stream.

For more information, see the Stream Command Line Client ``stream-client`` in the ``/bin`` directory of the
Continuuity Reactor SDK distribution.

For usage and documentation of options, run at the command line::

	$ stream-client --help


.. rst2pdf: PageBreak

Dataset HTTP API
================
The Dataset API allows you to interact with Datasets through HTTP. You can list, create, delete, and truncate Datasets. For details, see the Developer Guide:

.. rst2pdf: CutStart

.. only:: html

	`Continuuity Reactor Advanced Features, Datasets section <advanced.html#datasets-system>`__

.. only:: pdf

.. rst2pdf: CutStop

	`Continuuity Reactor Advanced Features, Datasets section <http://continuuity.com/docs/reactor/current/en/advanced.html#datasets-system>`__


Listing all Datasets
--------------------

You can list all Datasets in the Continuuity Reactor by issuing an HTTP GET request to the URL::

	GET <base-url>/data/datasets

The response body will contain a JSON-formatted list of the existing Datasets::

	{
	   "name":"continuuity.user.purchases",
	   "type":"com.continuuity.api.dataset.lib.ObjectStore",
	   "properties":{
	      "schema":"...",
	      "type":"..."
	   },
	   "datasetSpecs":{
	      ...
	   }
	 }

.. rst2pdf: PageBreak

Creating a Dataset
------------------

You can create a Dataset by issuing an HTTP PUT request to the URL::

	PUT <base-url>/data/datasets/<dataset-name>
  
with JSON-formatted name of the dataset type and properties in a body::

  {
     "typeName":"<type-name>",
     "properties":{<properties>}
  }


.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<dataset-name>``
     - Name of the new Dataset
   * - ``<type-name>``
     - Type of the new Dataset
   * - ``<properties>``
     - Dataset properties, map of String to String.

HTTP Responses
..............
.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - Requested Dataset was successfully created
   * - ``404 Not Found``
     - Requested Dataset type was not found
   * - ``409 Conflict``
     - Dataset with the same name already exists

Example
.......
.. list-table::
   :widths: 25 75
   :stub-columns: 1

   * - HTTP Request
     - ``PUT <base-url>/data/datasets/mydataset``
   * - Body
     - ``{"typeName":"com.continuuity.api.dataset.table.Table",`` ``"properties":{"ttl":"3600000"}}``
   * - Description
     - Creates a Dataset named "mydataset" of the type "table" and time-to-live property set to 1 hour

.. rst2pdf: PageBreak

Deleting a Dataset
------------------

You can delete a Dataset by issuing an HTTP DELETE request to the URL::

  DELETE <base-url>/data/datasets/<dataset-name>

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - Dataset was successfully deleted
   * - ``404 Not Found``
     - Dataset named ``<dataset-name>`` could not be found

Example
.......
.. list-table::
   :widths: 25 75
   :stub-columns: 1

   * - HTTP Request
     - ``DELETE <base-url>/data/datasets/mydataset``
   * - Description
     - Deletes the Dataset named "mydataset"

.. rst2pdf: PageBreak

Deleting all Datasets
---------------------

You can delete all Datasets by issuing an HTTP DELETE request to the URL::

  DELETE <base-url>/data/unrecoverable/datasets

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - All Datasets were successfully deleted

Truncating a Dataset
--------------------

You can truncate a Dataset by issuing an HTTP POST request to the URL::

  POST <base-url>/data/datasets/<dataset-name>/admin/truncate

This will clear the existing data from the Dataset. This cannot be undone.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - Dataset was successfully truncated

.. rst2pdf: PageBreak


Data HTTP API (Deprecated)
==========================

The Data API allows you to interact with Continuuity Reactor Tables (the core Datasets) through HTTP.
You can create Tables, truncate Tables, and read, write, modify, or delete data.

For Datasets other than Tables, you can truncate the Dataset using this API.

Creating a new Table
--------------------

To create a new table, issue an HTTP PUT method to the URL::

	PUT <base-url>/tables/<table-name>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be created

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the Table was either created or already exists
   * - ``409 Conflict``
     - A Dataset of a different type already exists with the given name

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/tables/streams/mytable``
   * - Description
     - Create a new Table named *mytable*

Comments
........
This will create a Table with the name given by ``<table-name>``.
Table names should only contain ASCII letters, digits and hyphens.
If a Table with the same name already exists, no error is returned,
and the existing Table remains in place.

However, if a Dataset of a different type exists with the same name—for example,
a key/value Table or ``KeyValueTable``—this call will return a ``409 Conflict`` error.

.. rst2pdf: PageBreak

Writing Data to a Table
-----------------------
To write to a table, send an HTTP PUT method to the table’s URI::

	PUT <base-url>/tables/<table-name>/rows/<row-key>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be written to
   * - ``<row-key>``
     - Row identifier

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the Table was successfully written to
   * - ``400 Bad Request``
     - The JSON String map is not well-formed or cannot be parsed as a map from String to String
   * - ``404 Not Found``
     - A Table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/tables/mytable/rows/status``
   * - Description
     - Write to the existing Table named *mytable* in a row identified as *status*

Comments
........
In the body of the request, you must specify the columns and values
that you want to write to the Table as a JSON String map. For example::

	{ "x":"y", "y":"a", "z":"1" }

This writes three columns named *x*, *y*, and *z* with values *y*, *a*, and *1*, respectively.

.. rst2pdf: PageBreak

Reading Data from a Table
-------------------------
To read data from a Table, address the row that you want to read directly
in an HTTP GET method to the table’s URI::

	GET <base-url>/tables/<table-name>/rows/<row-key>[?<column-identifier>]

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be read from
   * - ``<row-key>``
     - Row identifier
   * - ``<column-identifiers>``
     - An optional combination of attributes and values such as:
       ``start=<column-id> | stop=<column-id> | columns=<column-id>,<column-id>``

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the Table was successfully read from
   * - ``400 Bad Request``
     - The column list is not well-formed or cannot be parsed
   * - ``404 Not Found``
     - A Table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``GET <base-url>/tables/mytable/rows/status``
   * - Description
     - Read from an existing Table named *mytable*, a row identified as *status*

.. rst2pdf: PageBreak

Comments
........
The response will be a JSON String representing a map from column name to value.
For example, reading the row that was written in the `Writing Data to a Table`_, the response is::

	{"x":"y","y":"a","z":"1"}

If you are only interested in selected columns,
you can specify a list of columns explicitly or give a range of columns.

For example:

To return only columns *x* and *y*::

	GET ... /rows/<row-key>?columns=x,y

To return all columns equal to or greater than (inclusive) *c5*::

	GET ... /rows/<row-key>?start=c5

To return all columns less than (exclusive, not including) *c5*::

	GET ... /rows/<row-key>?stop=c5

To return all columns equal to or greater than (inclusive) *c2* and less than (exclusive, not including) *c5*::

	GET .../rows/<row-key>?start=c2&stop=c5


.. rst2pdf: PageBreak

Increment Data in a Table
-------------------------
You can perform an atomic increment of cells of a Table's row, and receive back the incremented values,
by issue an HTTP POST method to the row’s URL::

	POST <base-url>/tables/<table-name>/rows/<row-key>/increment

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be read from
   * - ``<row-key>``
     - Row identifier of row to be read

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event successfully incremented the row of the Table
   * - ``400 Bad Request``
     - The JSON String is not well-formed; or cannot be parsed as a map from a String to a Long;
       or one of the existing column values is not an 8-byte long value
   * - ``404 Not Found``
     - A table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mytable/rows/status/increment``
   * - Description
     - To increment the columns of *mytable*, in a row identified as *status*, by 1

Comments
........
In the body of the method, you must specify the columns and values that you want to increment them by
as a JSON map from Strings to Long numbers, such as::

	{ "x": 1, "y": 7 }

This HTTP call has the same effect as the corresponding Java Table Increment method.

If successful, the response contains a JSON String map from the column keys to the incremented values.

For example, if the existing value of column *x* was 4, and column *y* did not exist, then the response would be::

	{"x":5,"y":7}

Column *y* is newly created.

.. rst2pdf: PageBreak

Delete Data from a Table
------------------------
To delete from a table, submit an HTTP DELETE method::

	DELETE <base-url>/tables/<table-name>/rows/<row-key>[?<column-identifier>]

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be deleted from
   * - ``<row-key>``
     - Row identifier
   * - ``<column-identifiers>``
     - An optional combination of attributes and values such as::

       	start=<column-id> | stop=<column-id> | columns=<column-id>,<column-id>

..

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event successfully deleted the data of the Table
   * - ``404 Not Found``
     - A table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``DELETE <base-url>/tables/mytable/rows/status``
   * - Description
     - Deletes from an existing Table named *mytable*, a row identified as *status*

Comments
........
Similarly to `Reading Data from a Table`_, explicitly list the columns that you want to delete
by adding a parameter of the form ``?columns=<column-key,...>``.
See the examples under `Reading Data from a Table`_.

.. rst2pdf: PageBreak

Deleting Data from a Dataset
----------------------------

To clear a Dataset of all data, submit an HTTP POST request::

	POST <base-url>/datasets/<dataset-name>/truncate

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<dataset-name>``
     - Name of the Dataset to be truncated

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event successfully deleted the data of the Dataset
   * - ``404 Not Found``
     - A Dataset with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/datasets/mydataset/truncate``
   * - Description
     - Delete all of the data from an existing Dataset named *mydataset*

Comments
........
Note that this works not only for Tables but with other Datasets, including user-defined Custom Datasets.

.. rst2pdf: PageBreak

Encoding of Keys and Values
---------------------------

The URLs and JSON bodies of your HTTP requests contain row keys, column keys and values,
all of which are binary byte Arrays in the Java API.

You need to encode these binary keys and values as Strings in the URL and the JSON body
(the exception is the `Increment Data in a Table`_ method, which always interprets values as long integers).

The encoding parameter of the URL specifies the encoding used in both the URL and the JSON body.

For example, if you append a parameter ``encoding=hex`` to the request URL,
then all keys and values are interpreted as hexadecimal strings,
and the returned JSON from read requests also has keys and values encoded as hexadecimal string.

Be aware that the same encoding applies to all keys and values involved in a request.

For example, suppose you incremented table *counters*, row *a*, column *x* by 42::

	POST <base-url>/tables/counters/rows/a/increment {"x":42}

Now the value of column *x* is the 8-byte number 42. If you query for the value of this column::

	GET <base-url>/tables/counters/rows/a?columns=x

The returned JSON String map will contain a non-printable string for the value of column *x*::

	{"x":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000*"}

Note the Unicode escapes in the string, and the asterisk at the end (which is the character at code point 42).

To make this legible, you can specify hexadecimal notation in your request;
that will require that you also encode the row key
(*a*, encoded as *61*)
and the column key (*x*, encoded as *78*) in your request as hexadecimal::

	GET <base-url>/tables/counters/rows/61?columns=78&encoding=hex

The response now contains both the column key and the value as hexadecimal strings::

	{"78":"000000000000002a"}

The supported encodings are:

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Encoding
     - Description
   * - ``encoding=ascii``
     - Only ASCII characters are supported and are mapped to bytes one-to-one (Default)
   * - ``encoding=hex``
     - Hexadecimal strings. Example: the ASCII string ``a:b`` is represented as ``613A62``
   * - ``encoding=url``
     - URL encoding (also known as %-encoding or percent-encoding).
       URL-safe characters use ASCII-encoding, while other bytes values are escaped using a ``%`` sign.
       Example: the hexadecimal value ``613A62`` (ASCII string ``a:b``)
       is represented as the string ``a%3Ab``.
   * - ``encoding=base64``
     - URL-safe Base-64 encoding without padding.
       For more information, see `Internet RFC 2045 <http://www.ietf.org/rfc/rfc2045.txt>`_.
       Example: the hexadecimal value ``613A62`` is represented as the string ``YTpi``.

If you specify an encoding that is not supported, or you specify keys or values that cannot be decoded using that encoding, the request will return HTTP code ``400 Bad Request``.


Counter Values
--------------
Your Table values may frequently be counters (numbers), whereas the row and column keys might not be numbers.

In such cases, it is more convenient to represent your Table values as numeric strings,
by specifying ``counter=true``. For example::

	GET <base-url>/tables/counters/rows/a?columns=x&counter=true

The response now contains the column key as text and the row value as a numeric string::

	{"x":"42"}

Note that you can also specify the ``counter=true`` parameter when writing to a Table.
This allows you to specify values as numeric strings while using a different encoding for row and column keys.

Query HTTP API
==============

This interface supports submitting SQL queries over Datasets. Executing a query is asynchronous: 

- first, **submit** the query;
- then poll for the query's **status** until it is finished;
- once finished, retrieve the **result schema** and the **results**;
- finally, **close the query** to free the resources that it holds.

Submitting a Query
------------------
To submit a SQL query, post the query string to the ``queries`` URL::

  POST <base-url>/data/queries

The body of the request must contain a JSON string of the form::

  {
    "query": "<SQL-query-string>"
  }

where ``<SQL-query-string>`` is the actual SQL query.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query execution was successfully initiated, and the body will contain the query-handle
       used to identify the query in subsequent requests
   * - ``400 Bad Request``
     - The query is not well-formed or contains an error, such as a nonexistent table name.

Comments
........
If the query execution was successfully initiated, the body will contain a handle 
used to identify the query in subsequent requests::

	{ "handle":"<query-handle>" }

.. rst2pdf: PageBreak

Status of a Query
-----------------
The status of a query is obtained using a HTTP GET request to the query's URL::

  GET <base-url>/data/queries/<query-handle>

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query exists and the body contains its status
   * - ``404 Not Found``
     - The query handle does not match any current query.

Comments
........
If the query exists, the body will contain the status of its execution
and whether the query has a results set::

	{
	  "status":"<status-code>",
	  "hasResults":<boolean>
	 }

Status codes include ``INITIALIZED``, ``RUNNING``, ``FINISHED``, ``CANCELED``, ``CLOSED``,
``ERROR``, ``UNKNOWN``, and ``PENDING``.


Obtaining the Result Schema
---------------------------
If the query's status is ``FINISHED`` and it has results, you can obtain the schema of the results::

  GET <base-url>/data/queries/<query-handle>/schema

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query was successfully received and the query schema was returned in the body
   * - ``404 Not Found``
     - The query handle does not match any current query

Comments
........
The query's result schema is returned in a JSON body as a list of columns,
each given by its name, type and position; if the query has no result set, this list is empty::

	[
	  {"name":"<name>", "type":"<type>", "position":<int>},
	  ...
	]

The type of each column is a data type as defined in the `Hive language manual
<https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL>`_.


Retrieving Query Results
------------------------
Query results can be retrieved in batches after the query is finished, optionally specifying the batch
size in the body of the request::

  POST <base-url>/data/queries/<query-handle>/next

The body of the request can contain a JSON string specifying the batch size::

  {
    "size":<int>
  }

If the batch size is not specified, it defaults to 20.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the result of the query was returned in the body
   * - ``404 Not Found``
     - The query handle does not match any current query

Comments
........
The results are returned in a JSON body as a list of columns,
each given as a structure containing a list of column values.::

	[
	  { "columns": [ <value_1>, <value_2>, ..., ] },
	  ...
	]

The value at each position has the type that was returned in the result schema for that position.
For example, if the returned type was ``INT``, then the value will be an integer literal,
whereas for ``STRING`` or ``VARCHAR`` the value will be a string literal.

Repeat the query to retrieve subsequent results. If all results of the query have already 
been retrieved, then the returned list is empty. 

.. rst2pdf: PageBreak

Closing a Query
---------------
The query can be closed by issuing an HTTP DELETE against its URL::

  DELETE <base-url>/data/queries/<query-handle>

This frees all resources that are held by this query.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query was closed
   * - ``400 Bad Request``
     - The query was not in a state that could be closed; either wait until it is finished, or cancel it
   * - ``404 Not Found``
     - The query handle does not match any current query

Canceling a Query
-----------------
Execution of a query can be canceled before it is finished with an HTTP POST::

  POST <base-url>/data/queries/<query-handle>/cancel

After this, the query can only be closed.
<commit_msg>Doc changes related to explor endpoint refactoring<commit_after>.. :author: Continuuity, Inc.
   :version: 2.3.0
   :description: HTTP Interface to the Continuuity Reactor

=================================
Continuuity Reactor HTTP REST API
=================================

.. reST Editor: .. section-numbering::
.. reST Editor: .. contents::

.. rst2pdf: .. contents::
.. rst2pdf: config _templates/pdf-config
.. rst2pdf: stylesheets _templates/pdf-stylesheet
.. rst2pdf: build ../build-pdf/

Introduction
============

The Continuuity Reactor has an HTTP interface for a multitude of purposes:

- **Stream:** sending data events to a Stream, or to inspect the contents of a Stream
- **Dataset:** interacting with Datasets, Dataset Modules, and Dataset Types
- **Data:** interacting with Datasets (deprecated)
- **Query:** sending ad-hoc queries to Reactor Datasets
- **Procedure:** sending calls to a stored Procedure
- **Reactor Client:** deploying and managing Applications, and managing the life cycle of Flows,
  Procedures, MapReduce jobs, Workflows, and Custom Services
- **Logging:** retrieving Application logs
- **Metrics:** retrieving metrics for system and user Applications (user-defined metrics)
- **Monitor:** checking the status of various Reactor services, both System and Custom

**Note:** The HTTP interface binds to port ``10000``. This port cannot be changed.

Conventions
-----------

In this API, *client* refers to an external application that is calling the Continuuity Reactor using the HTTP interface.

In this API, *Application* refers to a user Application that has been deployed into the Continuuity Reactor.

All URLs referenced in this API have this base::

	http://<gateway>:10000/v2

where ``<gateway>`` is the URL of the Continuuity Reactor. The base URL is represented as::

	<base-url>

For example::

	PUT <base-url>/streams/<new-stream-id>

means
::

	PUT http://<gateway>:10000/v2/streams/<new-stream-id>
	

Text that are variables that you are to replace is indicated by a series of angle brackets (``< >``). For example::

	PUT <base-url>/streams/<new-stream-id>

indicates that—in addition to the ``<base-url>``—the text ``<new-stream-id>`` is a variable
and that you are to replace it with your value, perhaps in this case *mystream*::

	PUT <base-url>/streams/mystream

.. rst2pdf: PageBreak

Status Codes
------------

`Common status codes <http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html>`__ returned for all HTTP calls:


.. list-table::
   :widths: 10 30 60
   :header-rows: 1

   * - Code
     - Description
     - Explanation
   * - ``200``
     - ``OK``
     - The request returned successfully
   * - ``400``
     - ``Bad Request``
     - The request had a combination of parameters that is not recognized
   * - ``401``
     - ``Unauthorized``
     - The request did not contain an authentication token
   * - ``403``
     - ``Forbidden``
     - The request was authenticated but the client does not have permission
   * - ``404``
     - ``Not Found``
     - The request did not address any of the known URIs
   * - ``405``
     - ``Method Not Allowed``
     - A request was received with a method not supported for the URI
   * - ``409``
     - ``Conflict``
     - A request could not be completed due to a conflict with the current resource state
   * - ``500``
     - ``Internal Server Error``
     - An internal error occurred while processing the request
   * - ``501``
     - ``Not Implemented``
     - A request contained a query that is not supported by this API

**Note:** These returned status codes are not necessarily included in the descriptions of the API,
but a request may return any of these.


SSL Required for Sandboxed Continuuity Reactor
----------------------------------------------
When you interact with a Sandboxed Continuuity Reactor,
the Continuuity HTTP APIs require that you use SSL for the connection
and that you authenticate your request by sending your API key in an HTTP header::

	X-Continuuity-ApiKey: <api-key>

.. list-table::
   :widths: 15 85
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<api-key>``
     - Continuuity Reactor API key, obtained from an account at
       `Continuuity Accounts <http://accounts.continuuity.com>`__


Working with Reactor Security
-----------------------------
When working with a Reactor cluster with security enabled (``security.enabled=true`` in
``continuuity-site.xml``), all calls to the HTTP APIs must be authenticated.  Clients must first
obtain an access token from the authentication server (see the "Security" section of the
guide). In order to authenticate, all client requests must supply this access token in the
``Authorization`` header of the request::

   Authorization: Bearer <PASSWORD>

For Reactor-issued access tokens, the authentication scheme must always be ``Bearer``.


Stream HTTP API
===============
This interface supports creating Streams, sending events to a Stream, and reading single events from a Stream.

Streams may have multiple consumers (for example, multiple Flows), each of which may be a group of different agents (for example, multiple instances of a Flowlet).

In order to read events from a Stream, a client application must first obtain a consumer (group) id, which is then passed to subsequent read requests.


Creating a Stream
-----------------
A Stream can be created with an HTTP PUT method to the URL::

	PUT <base-url>/streams/<new-stream-id>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<new-stream-id>``
     - Name of the Stream to be created

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event either successfully created a Stream or the Stream already exists

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/streams/mystream``
   * - Description
     - Create a new Stream named *mystream*

Comments
........
- The ``<new-stream-id>`` should only contain ASCII letters, digits and hyphens.
- If the Stream already exists, no error is returned, and the existing Stream remains in place.

.. rst2pdf: PageBreak

Sending Events to a Stream
--------------------------
An event can be sent to a Stream by sending an HTTP POST method to the URL of the Stream::

	POST <base-url>/streams/<stream-id>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received
   * - ``404 Not Found``
     - The Stream does not exist

:Note: The response will always have an empty body

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream``
   * - Description
     - Send an event to the existing Stream named *mystream*

Comments
........
- The body of the request must contain the event in binary form.
- You can pass headers for the event as HTTP headers by prefixing them with the *stream-id*::

	<stream-id>.<property>:<string value>

  After receiving the request, the HTTP handler transforms it into a Stream event:

  #. The body of the event is an identical copy of the bytes
     found in the body of the HTTP post request.
  #. If the request contains any headers prefixed with the *stream-id*,
     the *stream-id* prefix is stripped from the header name and
     the header is added to the event.

.. rst2pdf: PageBreak

Reading Events from a Stream: Getting a Consumer-ID
---------------------------------------------------
Get a *Consumer-ID* for a Stream by sending an HTTP POST method to the URL::

	POST <base-url>/streams/<stream-id>/consumer-id

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

HTTP Responses
..............
.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and a new ``consumer-id`` was returned
   * - ``404 Not Found``
     - The Stream does not exist

Example
.......
.. list-table::
   :widths: 30 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream/consumer-id``
   * - Description
     - Request a ``Consumer-ID`` for the Stream named *mystream*

Comments
........
- Streams may have multiple consumers (for example, multiple Flows), 
  each of which may be a group of different agents (for example, multiple instances of a Flowlet).
- In order to read events from a Stream, a client application must
  first obtain a consumer (group) id, which is then passed to subsequent read requests.
- The ``Consumer-ID`` is returned in a response header and—for convenience—also in the body of the response::

	X-Continuuity-ConsumerId: <consumer-id>

  Once you have the ``Consumer-ID``, single events can be read from the Stream.

.. rst2pdf: PageBreak

Reading Events from a Stream: Using the Consumer-ID
---------------------------------------------------
A read is performed as an HTTP POST method to the URL::

	POST <base-url>/streams/<stream-id>/dequeue

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

The request must pass the ``Consumer-ID`` in a header of the form::

	X-Continuuity-ConsumerId: <consumer-id>

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the result of the read was returned
   * - ``204 No Content``
     - The Stream exists but it is either empty or the given ``Consumer-ID``
       has read all the events in the Stream
   * - ``404 Not Found``
     - The Stream does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream/dequeue``
   * - Description
     - Read the next event from an existing Stream named *mystream*

Comments
........
The read will always return the next event from the Stream that was inserted first and has not been read yet
(first-in, first-out or FIFO semantics). If the Stream has never been read from before, the first event will be read.

For example, in order to read the third event that was sent to a Stream,
two previous reads have to be performed after receiving the ``Consumer-ID``.
You can always start reading from the first event by getting a new ``Consumer-ID``.

The response will contain the binary body of the event in its body and a header for each header of the Stream event,
analogous to how you send headers when posting an event to the Stream::

	<stream-id>.<property>:<value>

.. rst2pdf: PageBreak

Truncating a Stream
-------------------
Truncation means the deletion of all events that were written to the Stream. 
This is permanent and cannot be undone.
A Stream can be truncated with an HTTP POST method to the URL::

	POST <base-url>/streams/<stream-id>/truncate

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The Stream was successfully truncated
   * - ``404 Not Found``
     - The Stream ``<stream-id>`` does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mystream/truncate``
   * - Description
     - Delete all events in the Stream named *mystream*

.. rst2pdf: PageBreak

Setting Time-To-Live Property of a Stream
------------------------------------------
The Time-To-Live (TTL) property governs how long an event is valid for consumption since 
it was written to the Stream.
The default TTL for all Streams is infinite, meaning that events will never expire.
The TTL property of a Stream can be changed with an HTTP PUT method to the URL::

	PUT <base-url>/streams/<stream-id>/config

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<stream-id>``
     - Name of an existing Stream

The new TTL value is passed in the request body as::

	{ "ttl" : <ttl-in-seconds> }

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<ttl-in-seconds>``
     - Number of seconds that an event will be valid for since ingested

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The stream TTL was changed successfully
   * - ``400 Bad Request``
     - The TTL value is not a non-negative integer
   * - ``404 Not Found``
     - The Stream does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/streams/mystream/config``

       with the new TTL value as a JSON string in the body::

	  { "ttl" : 86400 }
     
   * - Description
     - Change the TTL property of the Stream named *mystream* to 1 day


Reading Multiple Events
-----------------------
Reading multiple events is not supported directly by the Stream HTTP API,
but the command-line tool ``stream-client`` demonstrates how to view *all*, the *first N*, or the *last N* events in the Stream.

For more information, see the Stream Command Line Client ``stream-client`` in the ``/bin`` directory of the
Continuuity Reactor SDK distribution.

For usage and documentation of options, run at the command line::

	$ stream-client --help


.. rst2pdf: PageBreak

Dataset HTTP API
================
The Dataset API allows you to interact with Datasets through HTTP. You can list, create, delete, and truncate Datasets. For details, see the Developer Guide:

.. rst2pdf: CutStart

.. only:: html

	`Continuuity Reactor Advanced Features, Datasets section <advanced.html#datasets-system>`__

.. only:: pdf

.. rst2pdf: CutStop

	`Continuuity Reactor Advanced Features, Datasets section <http://continuuity.com/docs/reactor/current/en/advanced.html#datasets-system>`__


Listing all Datasets
--------------------

You can list all Datasets in the Continuuity Reactor by issuing an HTTP GET request to the URL::

	GET <base-url>/data/datasets

The response body will contain a JSON-formatted list of the existing Datasets::

	{
	   "name":"continuuity.user.purchases",
	   "type":"com.continuuity.api.dataset.lib.ObjectStore",
	   "properties":{
	      "schema":"...",
	      "type":"..."
	   },
	   "datasetSpecs":{
	      ...
	   }
	 }

.. rst2pdf: PageBreak

Creating a Dataset
------------------

You can create a Dataset by issuing an HTTP PUT request to the URL::

	PUT <base-url>/data/datasets/<dataset-name>
  
with JSON-formatted name of the dataset type and properties in a body::

  {
     "typeName":"<type-name>",
     "properties":{<properties>}
  }


.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<dataset-name>``
     - Name of the new Dataset
   * - ``<type-name>``
     - Type of the new Dataset
   * - ``<properties>``
     - Dataset properties, map of String to String.

HTTP Responses
..............
.. list-table::
   :widths: 25 75
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - Requested Dataset was successfully created
   * - ``404 Not Found``
     - Requested Dataset type was not found
   * - ``409 Conflict``
     - Dataset with the same name already exists

Example
.......
.. list-table::
   :widths: 25 75
   :stub-columns: 1

   * - HTTP Request
     - ``PUT <base-url>/data/datasets/mydataset``
   * - Body
     - ``{"typeName":"com.continuuity.api.dataset.table.Table",`` ``"properties":{"ttl":"3600000"}}``
   * - Description
     - Creates a Dataset named "mydataset" of the type "table" and time-to-live property set to 1 hour

.. rst2pdf: PageBreak

Deleting a Dataset
------------------

You can delete a Dataset by issuing an HTTP DELETE request to the URL::

  DELETE <base-url>/data/datasets/<dataset-name>

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - Dataset was successfully deleted
   * - ``404 Not Found``
     - Dataset named ``<dataset-name>`` could not be found

Example
.......
.. list-table::
   :widths: 25 75
   :stub-columns: 1

   * - HTTP Request
     - ``DELETE <base-url>/data/datasets/mydataset``
   * - Description
     - Deletes the Dataset named "mydataset"

.. rst2pdf: PageBreak

Deleting all Datasets
---------------------

You can delete all Datasets by issuing an HTTP DELETE request to the URL::

  DELETE <base-url>/data/unrecoverable/datasets

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - All Datasets were successfully deleted

Truncating a Dataset
--------------------

You can truncate a Dataset by issuing an HTTP POST request to the URL::

  POST <base-url>/data/datasets/<dataset-name>/admin/truncate

This will clear the existing data from the Dataset. This cannot be undone.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - Dataset was successfully truncated

.. rst2pdf: PageBreak


Data HTTP API (Deprecated)
==========================

The Data API allows you to interact with Continuuity Reactor Tables (the core Datasets) through HTTP.
You can create Tables, truncate Tables, and read, write, modify, or delete data.

For Datasets other than Tables, you can truncate the Dataset using this API.

Creating a new Table
--------------------

To create a new table, issue an HTTP PUT method to the URL::

	PUT <base-url>/tables/<table-name>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be created

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the Table was either created or already exists
   * - ``409 Conflict``
     - A Dataset of a different type already exists with the given name

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/tables/streams/mytable``
   * - Description
     - Create a new Table named *mytable*

Comments
........
This will create a Table with the name given by ``<table-name>``.
Table names should only contain ASCII letters, digits and hyphens.
If a Table with the same name already exists, no error is returned,
and the existing Table remains in place.

However, if a Dataset of a different type exists with the same name—for example,
a key/value Table or ``KeyValueTable``—this call will return a ``409 Conflict`` error.

.. rst2pdf: PageBreak

Writing Data to a Table
-----------------------
To write to a table, send an HTTP PUT method to the table’s URI::

	PUT <base-url>/tables/<table-name>/rows/<row-key>

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be written to
   * - ``<row-key>``
     - Row identifier

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the Table was successfully written to
   * - ``400 Bad Request``
     - The JSON String map is not well-formed or cannot be parsed as a map from String to String
   * - ``404 Not Found``
     - A Table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``PUT <base-url>/tables/mytable/rows/status``
   * - Description
     - Write to the existing Table named *mytable* in a row identified as *status*

Comments
........
In the body of the request, you must specify the columns and values
that you want to write to the Table as a JSON String map. For example::

	{ "x":"y", "y":"a", "z":"1" }

This writes three columns named *x*, *y*, and *z* with values *y*, *a*, and *1*, respectively.

.. rst2pdf: PageBreak

Reading Data from a Table
-------------------------
To read data from a Table, address the row that you want to read directly
in an HTTP GET method to the table’s URI::

	GET <base-url>/tables/<table-name>/rows/<row-key>[?<column-identifier>]

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be read from
   * - ``<row-key>``
     - Row identifier
   * - ``<column-identifiers>``
     - An optional combination of attributes and values such as:
       ``start=<column-id> | stop=<column-id> | columns=<column-id>,<column-id>``

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the Table was successfully read from
   * - ``400 Bad Request``
     - The column list is not well-formed or cannot be parsed
   * - ``404 Not Found``
     - A Table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``GET <base-url>/tables/mytable/rows/status``
   * - Description
     - Read from an existing Table named *mytable*, a row identified as *status*

.. rst2pdf: PageBreak

Comments
........
The response will be a JSON String representing a map from column name to value.
For example, reading the row that was written in the `Writing Data to a Table`_, the response is::

	{"x":"y","y":"a","z":"1"}

If you are only interested in selected columns,
you can specify a list of columns explicitly or give a range of columns.

For example:

To return only columns *x* and *y*::

	GET ... /rows/<row-key>?columns=x,y

To return all columns equal to or greater than (inclusive) *c5*::

	GET ... /rows/<row-key>?start=c5

To return all columns less than (exclusive, not including) *c5*::

	GET ... /rows/<row-key>?stop=c5

To return all columns equal to or greater than (inclusive) *c2* and less than (exclusive, not including) *c5*::

	GET .../rows/<row-key>?start=c2&stop=c5


.. rst2pdf: PageBreak

Increment Data in a Table
-------------------------
You can perform an atomic increment of cells of a Table's row, and receive back the incremented values,
by issue an HTTP POST method to the row’s URL::

	POST <base-url>/tables/<table-name>/rows/<row-key>/increment

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be read from
   * - ``<row-key>``
     - Row identifier of row to be read

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event successfully incremented the row of the Table
   * - ``400 Bad Request``
     - The JSON String is not well-formed; or cannot be parsed as a map from a String to a Long;
       or one of the existing column values is not an 8-byte long value
   * - ``404 Not Found``
     - A table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/streams/mytable/rows/status/increment``
   * - Description
     - To increment the columns of *mytable*, in a row identified as *status*, by 1

Comments
........
In the body of the method, you must specify the columns and values that you want to increment them by
as a JSON map from Strings to Long numbers, such as::

	{ "x": 1, "y": 7 }

This HTTP call has the same effect as the corresponding Java Table Increment method.

If successful, the response contains a JSON String map from the column keys to the incremented values.

For example, if the existing value of column *x* was 4, and column *y* did not exist, then the response would be::

	{"x":5,"y":7}

Column *y* is newly created.

.. rst2pdf: PageBreak

Delete Data from a Table
------------------------
To delete from a table, submit an HTTP DELETE method::

	DELETE <base-url>/tables/<table-name>/rows/<row-key>[?<column-identifier>]

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<table-name>``
     - Name of the Table to be deleted from
   * - ``<row-key>``
     - Row identifier
   * - ``<column-identifiers>``
     - An optional combination of attributes and values such as::

       	start=<column-id> | stop=<column-id> | columns=<column-id>,<column-id>

..

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event successfully deleted the data of the Table
   * - ``404 Not Found``
     - A table with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``DELETE <base-url>/tables/mytable/rows/status``
   * - Description
     - Deletes from an existing Table named *mytable*, a row identified as *status*

Comments
........
Similarly to `Reading Data from a Table`_, explicitly list the columns that you want to delete
by adding a parameter of the form ``?columns=<column-key,...>``.
See the examples under `Reading Data from a Table`_.

.. rst2pdf: PageBreak

Deleting Data from a Dataset
----------------------------

To clear a Dataset of all data, submit an HTTP POST request::

	POST <base-url>/datasets/<dataset-name>/truncate

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Parameter
     - Description
   * - ``<dataset-name>``
     - Name of the Dataset to be truncated

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event successfully deleted the data of the Dataset
   * - ``404 Not Found``
     - A Dataset with the given name does not exist

Example
.......
.. list-table::
   :widths: 20 80
   :stub-columns: 1

   * - HTTP Method
     - ``POST <base-url>/datasets/mydataset/truncate``
   * - Description
     - Delete all of the data from an existing Dataset named *mydataset*

Comments
........
Note that this works not only for Tables but with other Datasets, including user-defined Custom Datasets.

.. rst2pdf: PageBreak

Encoding of Keys and Values
---------------------------

The URLs and JSON bodies of your HTTP requests contain row keys, column keys and values,
all of which are binary byte Arrays in the Java API.

You need to encode these binary keys and values as Strings in the URL and the JSON body
(the exception is the `Increment Data in a Table`_ method, which always interprets values as long integers).

The encoding parameter of the URL specifies the encoding used in both the URL and the JSON body.

For example, if you append a parameter ``encoding=hex`` to the request URL,
then all keys and values are interpreted as hexadecimal strings,
and the returned JSON from read requests also has keys and values encoded as hexadecimal string.

Be aware that the same encoding applies to all keys and values involved in a request.

For example, suppose you incremented table *counters*, row *a*, column *x* by 42::

	POST <base-url>/tables/counters/rows/a/increment {"x":42}

Now the value of column *x* is the 8-byte number 42. If you query for the value of this column::

	GET <base-url>/tables/counters/rows/a?columns=x

The returned JSON String map will contain a non-printable string for the value of column *x*::

	{"x":"\u0000\u0000\u0000\u0000\u0000\u0000\u0000*"}

Note the Unicode escapes in the string, and the asterisk at the end (which is the character at code point 42).

To make this legible, you can specify hexadecimal notation in your request;
that will require that you also encode the row key
(*a*, encoded as *61*)
and the column key (*x*, encoded as *78*) in your request as hexadecimal::

	GET <base-url>/tables/counters/rows/61?columns=78&encoding=hex

The response now contains both the column key and the value as hexadecimal strings::

	{"78":"000000000000002a"}

The supported encodings are:

.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Encoding
     - Description
   * - ``encoding=ascii``
     - Only ASCII characters are supported and are mapped to bytes one-to-one (Default)
   * - ``encoding=hex``
     - Hexadecimal strings. Example: the ASCII string ``a:b`` is represented as ``613A62``
   * - ``encoding=url``
     - URL encoding (also known as %-encoding or percent-encoding).
       URL-safe characters use ASCII-encoding, while other bytes values are escaped using a ``%`` sign.
       Example: the hexadecimal value ``613A62`` (ASCII string ``a:b``)
       is represented as the string ``a%3Ab``.
   * - ``encoding=base64``
     - URL-safe Base-64 encoding without padding.
       For more information, see `Internet RFC 2045 <http://www.ietf.org/rfc/rfc2045.txt>`_.
       Example: the hexadecimal value ``613A62`` is represented as the string ``YTpi``.

If you specify an encoding that is not supported, or you specify keys or values that cannot be decoded using that encoding, the request will return HTTP code ``400 Bad Request``.


Counter Values
--------------
Your Table values may frequently be counters (numbers), whereas the row and column keys might not be numbers.

In such cases, it is more convenient to represent your Table values as numeric strings,
by specifying ``counter=true``. For example::

	GET <base-url>/tables/counters/rows/a?columns=x&counter=true

The response now contains the column key as text and the row value as a numeric string::

	{"x":"42"}

Note that you can also specify the ``counter=true`` parameter when writing to a Table.
This allows you to specify values as numeric strings while using a different encoding for row and column keys.

Query HTTP API
==============

This interface supports submitting SQL queries over Datasets. Executing a query is asynchronous: 

- first, **submit** the query;
- then poll for the query's **status** until it is finished;
- once finished, retrieve the **result schema** and the **results**;
- finally, **close the query** to free the resources that it holds.

Submitting a Query
------------------
To submit a SQL query, post the query string to the ``queries`` URL::

  POST <base-url>/data/explore/queries

The body of the request must contain a JSON string of the form::

  {
    "query": "<SQL-query-string>"
  }

where ``<SQL-query-string>`` is the actual SQL query.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query execution was successfully initiated, and the body will contain the query-handle
       used to identify the query in subsequent requests
   * - ``400 Bad Request``
     - The query is not well-formed or contains an error, such as a nonexistent table name.

Comments
........
If the query execution was successfully initiated, the body will contain a handle 
used to identify the query in subsequent requests::

	{ "handle":"<query-handle>" }

.. rst2pdf: PageBreak

Status of a Query
-----------------
The status of a query is obtained using a HTTP GET request to the query's URL::

  GET <base-url>/data/explore/queries/<query-handle>/status

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query exists and the body contains its status
   * - ``404 Not Found``
     - The query handle does not match any current query.

Comments
........
If the query exists, the body will contain the status of its execution
and whether the query has a results set::

	{
	  "status":"<status-code>",
	  "hasResults":<boolean>
	 }

Status codes include ``INITIALIZED``, ``RUNNING``, ``FINISHED``, ``CANCELED``, ``CLOSED``,
``ERROR``, ``UNKNOWN``, and ``PENDING``.


Obtaining the Result Schema
---------------------------
If the query's status is ``FINISHED`` and it has results, you can obtain the schema of the results::

  GET <base-url>/data/explore/queries/<query-handle>/schema

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query was successfully received and the query schema was returned in the body
   * - ``404 Not Found``
     - The query handle does not match any current query

Comments
........
The query's result schema is returned in a JSON body as a list of columns,
each given by its name, type and position; if the query has no result set, this list is empty::

	[
	  {"name":"<name>", "type":"<type>", "position":<int>},
	  ...
	]

The type of each column is a data type as defined in the `Hive language manual
<https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL>`_.


Retrieving Query Results
------------------------
Query results can be retrieved in batches after the query is finished, optionally specifying the batch
size in the body of the request::

  POST <base-url>/data/explore/queries/<query-handle>/next

The body of the request can contain a JSON string specifying the batch size::

  {
    "size":<int>
  }

If the batch size is not specified, it defaults to 20.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The event was successfully received and the result of the query was returned in the body
   * - ``404 Not Found``
     - The query handle does not match any current query

Comments
........
The results are returned in a JSON body as a list of columns,
each given as a structure containing a list of column values.::

	[
	  { "columns": [ <value_1>, <value_2>, ..., ] },
	  ...
	]

The value at each position has the type that was returned in the result schema for that position.
For example, if the returned type was ``INT``, then the value will be an integer literal,
whereas for ``STRING`` or ``VARCHAR`` the value will be a string literal.

Repeat the query to retrieve subsequent results. If all results of the query have already 
been retrieved, then the returned list is empty. 

.. rst2pdf: PageBreak

Closing a Query
---------------
The query can be closed by issuing an HTTP DELETE against its URL::

  DELETE <base-url>/data/explore/queries/<query-handle>

This frees all resources that are held by this query.

HTTP Responses
..............
.. list-table::
   :widths: 20 80
   :header-rows: 1

   * - Status Codes
     - Description
   * - ``200 OK``
     - The query was closed
   * - ``400 Bad Request``
     - The query was not in a state that could be closed; either wait until it is finished, or cancel it
   * - ``404 Not Found``
     - The query handle does not match any current query

Canceling a Query
-----------------
Execution of a query can be canceled before it is finished with an HTTP POST::

  POST <base-url>/data/explore/queries/<query-handle>/cancel

After this, the query can only be closed.

{'fraction_non_alphanumeric': 0.13629095601436267, 'fraction_numerical': 0.01733617594254937, 'mean_word_length': 2.6686734588864875, 'pattern_counts': {'":': 68, '<': 303, '<?xml version=': 0, '>': 303, 'https://': 2, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7231709', 'n_tokens_mistral': 22003, 'n_tokens_neox': 20383, 'n_words': 8852}