<commit_before>comp-pool
=========

This node.js REST service allows distribution of computing jobs (in javascript) to a hive of web clients and will allow users to send their own jobs to the hive once the multiple security issues are adressed. Methods to try, server-side, the integrity and correcteness of job results are also provided.

It has nothing to do with pool cleaning appliances. Yet.


###### History ######

It was firstly planned to be a Sinatra-based service, but the node.js ability to allow users to execute code server side made us to finally choose it.<commit_msg>Added design info to README<commit_after>comp-pool
=========

This node.js REST service allows distribution of computing jobs (in javascript) to a hive of web clients and will allow users to send their own jobs to the hive once the multiple security issues are adressed. Methods to try, server-side, the integrity and correcteness of job results are also provided.

It has nothing to do with pool cleaning appliances. Yet.

Api Design
-------------
We use a volunteer pull distributed computation model over a REST API. That means clients can pull (`GET`) `jobs`, job input `variables` and `POST` `results` back to the API. We offer the following resources:

### Jobs ###
A `job` object includes:
* a javascript function named `execute_function`
* an optional `validate_result` function to be executed server-side, by default every result is accepted, otherwise they are validated by this function
* a `variable_scheduler` function that selects `variable objects` (see below) from the `job`'s collection to serve to clients, returning a `variable id`
* associated `metadata` such as a `description` localized text and job `owner`in the form of a `metadata[description][locale]` and `metadata[owner]`. 

Jobs can be served directly using external REST ids or through a scheduler that organizes them for efficient processing in clients.

    POST /job
To send a new job object, expected to be compiled by [Google Caja](https://developers.google.com/caja/) to enhance client's security as suggested [here](http://stackoverflow.com/questions/23758472/closing-access-to-global-variables-javascript). One of the objectives is job posting to be as automated as possible while being secure.

    GET /job/:id
Gets a job by id

    GET /job/scheduler/:scheduler_id`
Uses a scheduler to retrieve a job, internal algorithms are abstracted through this interface

### Job Variables ###
An input `variable` object is an arbitrary javascript object associated to a `job` and is passed to the `execute_function` on each run.

    POST /job/:id/variable
Posts a new variable to be served to clients. Gets validated by the job's `validate_result` function if this function exists, otherwise is accepted by default.

    GET /job/:id/variable
Gets a variable through a variable scheduler, to be chained with the next call

    GET /job/:id/variable/:id
Gets a variable by id

### Job Result ###
A `result` object is an arbitrary javascript object associated to a `variable` the result of the computation over this `variable` object.

    `POST /job/:id/variable/:id/result`
Adds a new result associated to a job and job's variable


#### History ####

It was firstly planned to be a Sinatra-based service, but the node.js ability to allow users to execute code server side made us to finally choose it.
{'fraction_non_alphanumeric': 0.0745249406175772, 'fraction_numerical': 0.0023752969121140144, 'mean_word_length': 4.356120826709062, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2691432', 'n_tokens_mistral': 866, 'n_tokens_neox': 824, 'n_words': 479}