<commit_before>#%RAML 0.8

title: Ephemeral Container Service (ECS) (%ECS_VERSION%)
version: v99.999
documentation:
  - title: Overview
    content: |
      The Ephemeral Container Service (ECS) was born out of
      the [Cloudfeaster](https://github.com/simonsdave/cloudfeaster.git) project.
      [Docker](https://www.docker.com/) makes it easy to package
      short lived (aka ephemeral) tasks in a docker image and run
      those tasks using
      the [Docker Remote API](https://docs.docker.com/engine/reference/api/docker_remote_api/).
      ECS adds a service on top these building blocks that allows ephemeral tasks
      to be run on a cluster of [CoreOS](https://coreos.com/) machines using a RESTful API.

  - title: Security
    content: |
      ##### In-transit
      All communication with an ECS deployment is done over SSL/TLS.
      The ECS runs on a cluster of [CoreOS](https://coreos.com/) machines
      with a layer 5 (TCP) load balancer running in front of the cluster.
      Layer 5 load balancing is required because [nginx](http://nginx.org)
      is used to terminate SSL/TLS on each node in the cluster.

      Note - ECS nodes host both the api and docs domains on a single IP
      with 2 distinct SSL/TLS certificates. Clients making requests into an ECS
      deployment should support
      [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication) to ensure
      the correct SSL/TLS certificate is used.

      ##### At-rest
      The ECS does not persist any data.

      ##### Authentication
      All requests (except API doc requests) into an ECS deployment
      must pass BASIC authentication against a key & secret. Keys & secrets
      are configured when a deployment is created.
      ```ecsctl.sh``` is typically used to generate keys & secrets.
      Keys are [UUIDs](https://docs.python.org/2/library/uuid.html#uuid.uuid4).
      Secrets are are 128-bit random numbers generated using
      [/dev/urandom](http://en.wikipedia.org/wiki//dev/urandom)
      (via Python's [os.urandom()](https://docs.python.org/2/library/os.html#os-miscfunc)).
      After secrets are generated, only a
      hashed version of the secrets is written to disk.

      ##### Authorization
      The ECS is intended to be used a low level, utility service
      sitting behind a higher level abstraction that implements authorization.
      As such, as soon as a requestor is authenticated they are able to access
      any of ECS' endpoints.

  - title: Performance
    content: |
      In the API docs you'll notice a *Performance Expectations* section for
      endpoint operations. These performance expectations describe the expected
      time to perform the requested operation **after** the request has reached
      the service code which implements the operation. In order to estimate expected
      time to perform an operation from your infrastructure, record the time it
      takes to send repeated requests to the ```/_noop``` endpoint and
      add this ```/_noop``` time to the performance expectation described in the
      API docs.

  - title: Request and Rate Limiting
    content: |
      Each ECS node in the cluster implements a set of request and rate
      limiting rules for the ```docs``` and ```api``` domains.
      The rules are:

      | Rule               | ```docs``` | ```api``` |
      |--------------------|:----------:|:---------:|
      | req / sec / key    | N/A        | 5         |
      | req / sec / IP     | 10         | 10        |
      | connections / key  | N/A        | 25        |
      | connections / IP   | 10         | 5         |
      | max request size   | N/A        | 5K        |

  - title: Scaling
    content: |
      The ECS runs on a cluster of [CoreOS](https://coreos.com/) machines
      with a layer 5 load balancer running in front of the cluster.
      Each node in the cluster runs the ECS service and schedules tasks
      on that node.
      From an ECS point of view, there are no dependancies between nodes and
      as such the ECS cluster is intended to scale horizontally to accommodate
      a broad variety of workloads.

  - title: Endpoint Documentation Conventions
    content: |
      The remainder of this page contains detailed endpoint documentation.
      There are a few conventions used in the endpoint documentation that
      are worth describing.
      * by default ```ephemeral_container_service.py``` starts on
        ```http://127.0.0.1:8448``` in a development environment; in the
        endpoint samples you should notice endpoint URLs start with
        the environment varilable ```ECS_ENDPOINT```; the rational here
        is that after setting the ```ECS_ENDPOINT``` environment variable
        you should be able to copy and paste all the sample requests
        regardless of the ip/port on which ```ephemeral_container_service.py```
        is listening.
      * in the examples you'll see ```-u $KEY:$SECRET``` on cURL
        command line; the ```KEY``` & ```SECRET``` environment variables refer to
        the authentication key and secret described <a href="#Security">here</a>

/tasks:
  displayName: Tasks
  description: |
    The ```/tasks``` endpoint enables creation and execution of tasks
  post:
    description: |
      Create and execute a task.

      ##### Authentication
        * BASIC authentication using key and secret as described <a href="#Security">here</a>

      ##### Performance Expectations
        * 99% of the time &lt;&nbsp;nnn ms

      ##### Example
      Below is a "hello world" style example of how to run a super simple task
      against a public repo.

      ```bash
      >cat echo.json
      {
        "docker_image": "ubuntu",
        "tag": "latest",
        "cmd": [
          "echo",
          "hello world!!!"
        ]
      }
      >curl \
        -s \
        -u $KEY:$SECRET \
        -X POST \
        -H "Content-Type: application/json" \
        --data-binary @echo.json \
        $ECS_ENDPOINT/v1.0/tasks | \
        jq
      {
        "stdout": "aGVsbG8gd29ybGQhISEK",
        "stderr": "",
        "exitCode": 0
      }
      >echo aGVsbG8gd29ybGQhISEK | base64 --decode
      'hello world!!!\n'
      >
      ```

      The json doc below illustrates how the hello world example request
      payload is modified if the task needs to be run against a private repo.

      ```json
      {
        "docker_image": "ubuntu",
        "tag": "latest",
        "cmd": [
          "echo",
          "hello world!!!"
        ],
        "creds": {
          "username": "something",
          "email": "<EMAIL>",
          "password": "password"
        }
      }
      ```

    responses:
      200:
        description:
          Success!
      400:
        description:
          Bad request.
      401:
        description:
          Authentication failed.
      404:
        description:
          Docker image not found.
      503:
        description:<commit_msg>add 413 response code to /tasks POST in API docs<commit_after>#%RAML 0.8

title: Ephemeral Container Service (ECS) (%ECS_VERSION%)
version: v99.999
documentation:
  - title: Overview
    content: |
      The Ephemeral Container Service (ECS) was born out of
      the [Cloudfeaster](https://github.com/simonsdave/cloudfeaster.git) project.
      [Docker](https://www.docker.com/) makes it easy to package
      short lived (aka ephemeral) tasks in a docker image and run
      those tasks using
      the [Docker Remote API](https://docs.docker.com/engine/reference/api/docker_remote_api/).
      ECS adds a service on top these building blocks that allows ephemeral tasks
      to be run on a cluster of [CoreOS](https://coreos.com/) machines using a RESTful API.

  - title: Security
    content: |
      ##### In-transit
      All communication with an ECS deployment is done over SSL/TLS.
      The ECS runs on a cluster of [CoreOS](https://coreos.com/) machines
      with a layer 5 (TCP) load balancer running in front of the cluster.
      Layer 5 load balancing is required because [nginx](http://nginx.org)
      is used to terminate SSL/TLS on each node in the cluster.

      Note - ECS nodes host both the api and docs domains on a single IP
      with 2 distinct SSL/TLS certificates. Clients making requests into an ECS
      deployment should support
      [SNI](https://en.wikipedia.org/wiki/Server_Name_Indication) to ensure
      the correct SSL/TLS certificate is used.

      ##### At-rest
      The ECS does not persist any data.

      ##### Authentication
      All requests (except API doc requests) into an ECS deployment
      must pass BASIC authentication against a key & secret. Keys & secrets
      are configured when a deployment is created.
      ```ecsctl.sh``` is typically used to generate keys & secrets.
      Keys are [UUIDs](https://docs.python.org/2/library/uuid.html#uuid.uuid4).
      Secrets are are 128-bit random numbers generated using
      [/dev/urandom](http://en.wikipedia.org/wiki//dev/urandom)
      (via Python's [os.urandom()](https://docs.python.org/2/library/os.html#os-miscfunc)).
      After secrets are generated, only a
      hashed version of the secrets is written to disk.

      ##### Authorization
      The ECS is intended to be used a low level, utility service
      sitting behind a higher level abstraction that implements authorization.
      As such, as soon as a requestor is authenticated they are able to access
      any of ECS' endpoints.

  - title: Performance
    content: |
      In the API docs you'll notice a *Performance Expectations* section for
      endpoint operations. These performance expectations describe the expected
      time to perform the requested operation **after** the request has reached
      the service code which implements the operation. In order to estimate expected
      time to perform an operation from your infrastructure, record the time it
      takes to send repeated requests to the ```/_noop``` endpoint and
      add this ```/_noop``` time to the performance expectation described in the
      API docs.

  - title: Request and Rate Limiting
    content: |
      Each ECS node in the cluster implements a set of request and rate
      limiting rules for the ```docs``` and ```api``` domains.
      The rules are:

      | Rule               | ```docs``` | ```api``` |
      |--------------------|:----------:|:---------:|
      | req / sec / key    | N/A        | 5         |
      | req / sec / IP     | 10         | 10        |
      | connections / key  | N/A        | 25        |
      | connections / IP   | 10         | 5         |
      | max request size   | N/A        | 5K        |

  - title: Scaling
    content: |
      The ECS runs on a cluster of [CoreOS](https://coreos.com/) machines
      with a layer 5 load balancer running in front of the cluster.
      Each node in the cluster runs the ECS service and schedules tasks
      on that node.
      From an ECS point of view, there are no dependancies between nodes and
      as such the ECS cluster is intended to scale horizontally to accommodate
      a broad variety of workloads.

  - title: Endpoint Documentation Conventions
    content: |
      The remainder of this page contains detailed endpoint documentation.
      There are a few conventions used in the endpoint documentation that
      are worth describing.
      * by default ```ephemeral_container_service.py``` starts on
        ```http://127.0.0.1:8448``` in a development environment; in the
        endpoint samples you should notice endpoint URLs start with
        the environment varilable ```ECS_ENDPOINT```; the rational here
        is that after setting the ```ECS_ENDPOINT``` environment variable
        you should be able to copy and paste all the sample requests
        regardless of the ip/port on which ```ephemeral_container_service.py```
        is listening.
      * in the examples you'll see ```-u $KEY:$SECRET``` on cURL
        command line; the ```KEY``` & ```SECRET``` environment variables refer to
        the authentication key and secret described <a href="#Security">here</a>

/tasks:
  displayName: Tasks
  description: |
    The ```/tasks``` endpoint enables creation and execution of tasks
  post:
    description: |
      Create and execute a task.

      ##### Authentication
        * BASIC authentication using key and secret as described <a href="#Security">here</a>

      ##### Performance Expectations
        * 99% of the time &lt;&nbsp;nnn ms

      ##### Example
      Below is a "hello world" style example of how to run a super simple task
      against a public repo.

      ```bash
      >cat echo.json
      {
        "docker_image": "ubuntu",
        "tag": "latest",
        "cmd": [
          "echo",
          "hello world!!!"
        ]
      }
      >curl \
        -s \
        -u $KEY:$SECRET \
        -X POST \
        -H "Content-Type: application/json" \
        --data-binary @echo.json \
        $ECS_ENDPOINT/v1.0/tasks | \
        jq
      {
        "stdout": "aGVsbG8gd29ybGQhISEK",
        "stderr": "",
        "exitCode": 0
      }
      >echo aGVsbG8gd29ybGQhISEK | base64 --decode
      'hello world!!!\n'
      >
      ```

      The json doc below illustrates how the hello world example request
      payload is modified if the task needs to be run against a private repo.

      ```json
      {
        "docker_image": "ubuntu",
        "tag": "latest",
        "cmd": [
          "echo",
          "hello world!!!"
        ],
        "creds": {
          "username": "something",
          "email": "<EMAIL>",
          "password": "password"
        }
      }
      ```

    responses:
      200:
        description:
          Success!
      400:
        description:
          Bad request.
      401:
        description:
          Authentication failed.
      404:
        description:
          Docker image not found.
      413:
        description:
          Request body too big.
          Max request size is 5K.
          See <a href="#Request-and-Rate-Limiting">this</a> for details.
      503:
        description:
{'fraction_non_alphanumeric': 0.09648625672995183, 'fraction_numerical': 0.009846982147917257, 'mean_word_length': 2.1723595505617976, 'pattern_counts': {'":': 26, '<': 15, '<?xml version=': 0, '>': 23, 'https://': 18, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1957094', 'n_tokens_mistral': 3948, 'n_tokens_neox': 3705, 'n_words': 1530}