<commit_before>Introduction
===============

Overview
---------

Shelloid is an open source IoT-ready real-time big data web application platform built using Node.js and Clojure that contains integrated stream query capability. Shelloid allows developers to easily integrate events from IoT sources such as the Nest thermostat by annotating functions with simple stream query specifications.

The following code illustrates this point

.. code-block:: javascript

	/**
	@stream.sdl select: ambient_temperature_f, name_long;
				from: nest.thermostat;
				where: ambient_temperature_f >= %threshold%
	@stream.init highTemperatureInit
	@stream.edgeTriggered
	*/
	exports.highTemperature = function(data, streamInst, user){
		console.log("highTemperature Data: ", data, " for user id ", user);
		for(var k in streamInst.pendingRequests){
			var pending = streamInst.pendingRequests[k];
			pending.send(data);
		}
	}

	exports.highTemperatureInit = function(streamDef, done){
		var instance = streamDef.newInstance({threshold:80}, 
		"global");
		done();
	}

	
In the above code snippets doc-style comments are used to succinctly specify annotations supported by Shelloid. If you prefer not to use comments for such a purpose, a direct style is also supported:

.. code-block:: javascript

	exports.highTemperature = 
	{
		annotations:{
			stream: 
			{
				sdl:"select: ambient_temperature_f, name_long;" +
					"from: nest.thermostat;" +
					"where: ambient_temperature_f >= %threshold%",
				init: highTemperatureInit,
				edgeTriggered: true
			}
		},
		fn: highTemperature
	}
	function highTemperature(data, streamInst, user){
		console.log("highTemperature Data: ", data, " for user id ", user);
		for(var k in streamInst.pendingRequests){
			var pending = streamInst.pendingRequests[k];
			pending.res.send(data);
		}	
	}
	function highTemperatureInit(streamDef, done){
		var instance = streamDef.newInstance({threshold:80}, 
		"global");
		done();
	}
	
	
In the above code snippet the "highTemperature" function defines a real-time IoT data stream owing to its associated stream annotations. The "stream.sdl" (sdl stands for stream definition language) defines the stream characteristics in a syntax remindful of the SQL. The "stream.init" specifies an initialization function, which in this case, creates a global stream instance. Shelloid will process the stream query in the background invoke the "highTemperature" function when the query condition holds true.

Shelloid supports special edge-triggered stream queries (specified with edgeTriggered annotation). An edge triggered stream handler is invoked if and only if there is a change in the query result from the previous query.

Each stream has an associated URL, similar to the way URLs are associated with controllers.  An HTTP request to this URL essentially supplies the caller with the stream query result. The HTTP response to stream URLs contains a 'tag' field which can be provided back as query parameter.  Whenever there is change in the query result the tag maintained by the server changes. If tag field is specified then the request will return only if server's tag changes from the specified tag. If provided tag is same as the server's data tag for the stream instance, Shelloid will hold the request (long polling style) until the stream event fires - upon which the stream handler can list the pending requests and process them one by one (support for websocket-based streaming is in progress). 

Shelloid stack is built using cutting-edge technologies such as Node.js (for the web tier), MongoDB (analytics DB), and Clojure (real-time stream computations). Real-time events from raw data can be generating using simple, declarative SQL-like stream queries. 

With its stream query annotations, Shelloid enables web applications to easily generate and process real-time events from IoT sources such as the Nest thermostat, Fitbit/Jawbone fitness bands, and many more. Our vision is to simplify the development of secure and robust IoT-enabled web applications and web services, improving programmer productivity and enabling quick time-to-market. 

Shelloid takes care of the infrastructure logic and lets you focus on your business logic, leading to quick time to market for your business-critical applications. Shelloid is open sourced under LGPL license, allowing you to run your commercial closed source applications on the top of it.


Node - Clojure integration
--------------------------

Shelloid attempts to bring out the best of two cool modern programming platforms: Node.js and Clojure. Node.js is great for web and real-time. Clojure is great for concurrent computations. So, why not let the Node.js handle the web requests and real-time messaging and use Clojure when there is a heavy computation at hand? Shelloid does just that.

Shelloid is essentially a Node.js web application server with integrated Clojure-based compute service which runs as a separate process. Shelloid takes care of all the integration details. While the Clojure compute service is executing a heavy computation, Node.js event loop is freed up to process other requests.

The following code snippet illustrates how the integration works. In the code, we define a compute service named add in Clojure. From Node.js this service is invoked by calling sh.ccs.add function (sh stands for shelloid, ccs for Clojure compute service). This results in parameters being passed to the CCS process and the Clojure add service function being executed. The result of the Clojure function is passed back to Node.js and the callback is invoked with err message if any and the result value. After passing off the computation to the Clojure, Node.js event loop is freed up to execute other requests.

Node.js 

.. code-block:: javascript

	sh.ccs.add(100, 200, function(err, r){
	console.log("Result: " + r);
	});


Clojure

.. code-block:: clojure

	(service add [a b]
		(+ a b)
	)

Clojure compute service (CCS) requires `Leiningen <http://leiningen.org>`_ to be installed somewhere in the system path.

Please Note: More documentation is on the way. Please bear with us for couple of weeks!

Open, extensible architecture
-----------------------------

Shelloid has an open, extensible architecture. While open sourcing is great for the users, Shelloid goes beyond merely being open source. Shelloid has a open architecture created from ground up, allowing you to easily write extensions that can modify the behaviour of the engine to suit your needs. Many of Shelloid's  built-in features are themselves built as extensions, leading to a relatively small and robust core.

Our vision is to simplify the development of secure and robust web applications and services, improving programmer productivity and enabling quick time-to-market. Shelloid takes care of the infrastructure logic and lets you focus on your business logic, leading to quick time to market for your business-critical applications. Shelloid is open sourced under LGPL license, allowing you to run your commercial closed source applications on the top of it.


Key features
-------------

* Integrated Clojure compute service.
* Use of declarative annotations instead of writing code for many useful functions.
* Configurable automatic restarting of the server in case of changes to the application code or unrecoverable errors.
* Built in authentication (via passport.js) - requires only a single authentication function to be written.
* Currently supports local authentication as well as Google, Facebook, Twitter authentications out of the box.
* Built-in login session management.
* Built-in role-based access control with roles attached to controllers via annotations.
* Custom authentication, e.g, for API implementations that is attached to routes via annotations.
* Supports specification-based verification of API requests/responses. Simple API specification which is automatically checked against requests for enhanced security, robustness. The application code will be cleaner owing to lesser checks required.
* Built in cluster support by setting a single configuration flag. Built-in logging with cluster support.
* Simplified DB API with built-in connection pooling.
* Named SQL query strings can be declared as annotations and executed using the name.
* Built-in simulator for controlled functional testing of the application/controller logic (work in progress). Simulator will allow specification and verification of temporal properties, controlled flow of time, etc.
* Support for easily configurable UI themes.
* UI themes can be associated with domains, i.e., depending on the domain by which the site is accessed a separate set of files/views can be served. Note that, at the moment, controllers are shared across domains. This results in a limited support for virtual hosting.
* Built in proper error and exception handling that takes care of sending error responses and freeing DB connections.
* Built in simple and versatile sequencing API that avoids callback hell and results in readily understandable code.
* Simple config specification for allowing cross-origin requests (implementation complying with CORS standard).
* Auto detection of the current node of execution based on specified node names to IP/hostname mapping - useful for distributed and cloud deployments.
* And, more being added every day!
<commit_msg>stream auth<commit_after>Introduction
===============

Overview
---------

Shelloid is an open source IoT-ready real-time big data web application platform built using Node.js and Clojure that contains integrated stream query capability. Shelloid allows developers to easily integrate events from IoT sources such as the Nest thermostat by annotating functions with simple stream query specifications.

The following code illustrates this point

.. code-block:: javascript

	/**
	@stream.sdl select: ambient_temperature_f, name_long;
				from: nest.thermostat;
				where: ambient_temperature_f >= %threshold%
	@stream.init highTemperatureInit
	@stream.auth highTemperatureAuth
	@stream.edgeTriggered
	*/
	exports.highTemperature = function(data, streamInst, user){
		console.log("highTemperature Data: ", data, " for user id ", user);
	}
	exports.highTemperatureAuth = function(req, streamInst, user, done){
		done(true);
	}
	exports.highTemperatureInit = function(streamDef, done){
		var instance = streamDef.newInstance({threshold:80}, 
		"global");
		done();
	}

	
In the above code snippets doc-style comments are used to succinctly specify annotations supported by Shelloid. If you prefer not to use comments for such a purpose, a direct style is also supported:

.. code-block:: javascript

	exports.highTemperature = 
	{
		annotations:{
			stream: 
			{
				sdl:"select: ambient_temperature_f, name_long;" +
					"from: nest.thermostat;" +
					"where: ambient_temperature_f >= %threshold%",
				init: highTemperatureInit,
				edgeTriggered: true,
				auth: highTemperatureAuth
			}
		},
		fn: highTemperature
	}
	function highTemperature(data, streamInst, user){
		console.log("highTemperature Data: ", data, " for user id ", user);
		for(var k in streamInst.pendingRequests){
			var pending = streamInst.pendingRequests[k];
			pending.res.send(data);
		}	
	}
	function highTemperatureInit(streamDef, done){
		var instance = streamDef.newInstance({threshold:80}, 
		"global");
		done();
	}

	function highTemperatureAuth(req, streamInst, user, done){
		done(true);
	}
	
	
In the above code snippet the "highTemperature" function defines a real-time IoT data stream owing to its associated stream annotations. The "stream.sdl" (sdl stands for stream definition language) defines the stream characteristics in a syntax remindful of the SQL. The "stream.init" specifies an initialization function, which in this case, creates a global stream instance. Shelloid will process the stream query in the background invoke the "highTemperature" function when the query condition holds true.

Shelloid supports special edge-triggered stream queries (specified with edgeTriggered annotation). An edge triggered stream handler is invoked if and only if there is a change in the query result from the previous query.

Each stream has an associated URL, similar to the way URLs are associated with controllers.  An HTTP request to this URL essentially supplies the caller with the stream query result. The HTTP response to stream URLs contains a 'tag' field which can be provided back as query parameter.  Whenever there is change in the query result the tag maintained by the server changes. If tag field is specified then the request will return only if server's tag changes from the specified tag. If provided tag is same as the server's data tag for the stream instance, Shelloid will hold the request (long polling style) until the stream event fires (support for websocket-based streaming is in progress). 

The stream.auth annotation is used for specifying custom authorization function for stream URL requests (note that basic authorization can be achieved using standard authorization modules, e.g., Shelloid has a built-in RBAC module).

Shelloid stack is built using cutting-edge technologies such as Node.js (for the web tier), MongoDB (analytics DB), and Clojure (real-time stream computations). Real-time events from raw data can be generating using simple, declarative SQL-like stream queries. 

With its stream query annotations, Shelloid enables web applications to easily generate and process real-time events from IoT sources such as the Nest thermostat, Fitbit/Jawbone fitness bands, and many more. Our vision is to simplify the development of secure and robust IoT-enabled web applications and web services, improving programmer productivity and enabling quick time-to-market. 

Shelloid takes care of the infrastructure logic and lets you focus on your business logic, leading to quick time to market for your business-critical applications. Shelloid is open sourced under LGPL license, allowing you to run your commercial closed source applications on the top of it.


Node - Clojure integration
--------------------------

Shelloid attempts to bring out the best of two cool modern programming platforms: Node.js and Clojure. Node.js is great for web and real-time. Clojure is great for concurrent computations. So, why not let the Node.js handle the web requests and real-time messaging and use Clojure when there is a heavy computation at hand? Shelloid does just that.

Shelloid is essentially a Node.js web application server with integrated Clojure-based compute service which runs as a separate process. Shelloid takes care of all the integration details. While the Clojure compute service is executing a heavy computation, Node.js event loop is freed up to process other requests.

The following code snippet illustrates how the integration works. In the code, we define a compute service named add in Clojure. From Node.js this service is invoked by calling sh.ccs.add function (sh stands for shelloid, ccs for Clojure compute service). This results in parameters being passed to the CCS process and the Clojure add service function being executed. The result of the Clojure function is passed back to Node.js and the callback is invoked with err message if any and the result value. After passing off the computation to the Clojure, Node.js event loop is freed up to execute other requests.

Node.js 

.. code-block:: javascript

	sh.ccs.add(100, 200, function(err, r){
	console.log("Result: " + r);
	});


Clojure

.. code-block:: clojure

	(service add [a b]
		(+ a b)
	)

Clojure compute service (CCS) requires `Leiningen <http://leiningen.org>`_ to be installed somewhere in the system path.

Please Note: More documentation is on the way. Please bear with us for couple of weeks!

Open, extensible architecture
-----------------------------

Shelloid has an open, extensible architecture. While open sourcing is great for the users, Shelloid goes beyond merely being open source. Shelloid has a open architecture created from ground up, allowing you to easily write extensions that can modify the behaviour of the engine to suit your needs. Many of Shelloid's  built-in features are themselves built as extensions, leading to a relatively small and robust core.

Our vision is to simplify the development of secure and robust web applications and services, improving programmer productivity and enabling quick time-to-market. Shelloid takes care of the infrastructure logic and lets you focus on your business logic, leading to quick time to market for your business-critical applications. Shelloid is open sourced under LGPL license, allowing you to run your commercial closed source applications on the top of it.


Key features
-------------

* Integrated Clojure compute service.
* Use of declarative annotations instead of writing code for many useful functions.
* Configurable automatic restarting of the server in case of changes to the application code or unrecoverable errors.
* Built in authentication (via passport.js) - requires only a single authentication function to be written.
* Currently supports local authentication as well as Google, Facebook, Twitter authentications out of the box.
* Built-in login session management.
* Built-in role-based access control with roles attached to controllers via annotations.
* Custom authentication, e.g, for API implementations that is attached to routes via annotations.
* Supports specification-based verification of API requests/responses. Simple API specification which is automatically checked against requests for enhanced security, robustness. The application code will be cleaner owing to lesser checks required.
* Built in cluster support by setting a single configuration flag. Built-in logging with cluster support.
* Simplified DB API with built-in connection pooling.
* Named SQL query strings can be declared as annotations and executed using the name.
* Built-in simulator for controlled functional testing of the application/controller logic (work in progress). Simulator will allow specification and verification of temporal properties, controlled flow of time, etc.
* Support for easily configurable UI themes.
* UI themes can be associated with domains, i.e., depending on the domain by which the site is accessed a separate set of files/views can be served. Note that, at the moment, controllers are shared across domains. This results in a limited support for virtual hosting.
* Built in proper error and exception handling that takes care of sending error responses and freeing DB connections.
* Built in simple and versatile sequencing API that avoids callback hell and results in readily understandable code.
* Simple config specification for allowing cross-origin requests (implementation complying with CORS standard).
* Auto detection of the current node of execution based on specified node names to IP/hostname mapping - useful for distributed and cloud deployments.
* And, more being added every day!

{'fraction_non_alphanumeric': 0.05825344991046034, 'fraction_numerical': 0.0010534077741493732, 'mean_word_length': 4.470181503889369, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3914263', 'n_tokens_mistral': 4667, 'n_tokens_neox': 4305, 'n_words': 2531}