<commit_before># FIWARE IoT Agent Framework

## Index

* [Overview](#overview)
* [Usage](#usage)
* [IoT Library testing](#librarytesting)
* [Configuration](#configuration)
* [Device Provisioning API](#provisioningapi)
* [Configuration API](#configurationapi)
* [Secured access to the Context Broker](#securedaccess)
* [Development Documentation](#development)

## <a name="overview"/> Overview
### Description
This project aims to provide a node.js module to enable IoT Agent developers to build custom agents for their devices that can 
easily connect to NGSI Context Brokers (such as [Orion](https://github.com/telefonicaid/fiware-orion) ). 

An IoT Agent is a component that lets groups of devices send their data to and be managed from a FIWARE NGSI Context Broker using their own native protocols. IoT Agents should also be able to deal with security aspects of the fiware platform (authentication and authorization of the channel) and provide other common services to the device programmer.

### Device to NGSI Mapping
Each Device will be mapped as an Entity associated to a Context Provider: the Device Id will be mapped by default to the entity ID and the type of the entity will be selected by the IoT Agent in a protocol-dependent way (e.g: with different URLs for different types). Both the name and type will be configurable by the user, either by type configuration or with the device preprovisioning).

Each of the measures obtained from the device should be mapped to a different attribute. The name and type of the attribute will be configured by the user (globally for all the types in the IoT Agent configuration or in a per device basis preprovisioning the devices). Device measures can have two different behaviors:

* **Active attributes**: are measures that are pushed from the device to the IoT agent. This measure changes will be sent to the Context Broker as updateContext requests over the device entity. NGSI queries to the context broker will be resolved in the Broker database.
* **Lazy attributes**: some sensors will be passive, and will wait for the IoT Agent to request for data. For those measures, the IoT Agent will register itself in the Context Broker as a Context Provider (for all the lazy measures of that device), so if any component asks the Context Broker for the value of that sensor, its request will be redirected to the IoT Agent (that behaves as a NGSI10 Context Provider). 

The following sequence diagram shows the different NGSI interactions an IoT Agent makes with the Context Broker, explained in the following subsections (using the example of a OMA Lightweight M2M device).

![General ](https://raw.github.com/dmoranj/iotagent-node-lib/develop/img/ngsiInteractions.png "NGSI Interactions")

Be aware that the IoT Agents are only required to support NGSI10 operations `updateContext` and `queryContext` in their standard formats (both in XML and JSON) but will not answer to NGSI9 operations (or NGSI convenience operations of any kind).

#### Registration
Whenever a device is registered, the IoT Agent reads the device's entity information from the request or, it that information is not in the request, from the default values for that type of device. Among this information, theres should be the list of device attributes that will be considered lazy (or passive). With this information, the IoT Agent sends a new `registerContext` request to the Context Broker, registering itself as ContextProvider of all the lazy attributes for the device's entity. The `registrationId` is then stored along the other device information inside the IoT Agent device regitry.

As NGSI9 does not allow the context registrations to be removed, when the device is removed from the IoT Agent, the registration is updated to an expiration date of 1s, so it is effectively disabled. Once it has been disabled, the device is removed from the IoT Agent's internal registry.

#### Lazy attributes
When a request for data from a lazy attribute arrives to the Context Broker, it forwards the request to the Context Provider of that entity, in this case the IoT Agent. The IoT Agent will in turn ask the device for the information needed, transform that information to a NSGI format and return it to the Context Broker. The latter will the forward the response to the caller, transparently.

#### Commands
Commands are modelled as updates over a lazy attribute. As in the case of the lazy attributes, updates over a command will be forwarded by the Context Broker to the IoT Agent, that will in turn interact with the device to perform the requested action. Parameters for the command will be passed inside the command value.

There are two differences with the lazy attributes:
* First of all, for every command defined in a device, two new attributes are created in the entity with the same name as the command but with a prefix:
	* '_status': this attribute reflect the current execution status of the command. When a command request is issued by the Context Broker, the IoT Agent library generates this attribute with 'PENDING' value. The value of this attribute will be changed each time a command error or result is issued to the IoT Agent.
	* '_result': this attribute reflect the result of the execution of the defined command.

* Commands can also be updated when new information about its execution arrives to the agent. This information will be mapped to the command's utility attributes `_status` and `_result` leaving alone the command attribute itself. The values for this attributes are stored locally in the Context Broker (instead of being redirected with the Context Provider operations).

#### Active attributes
Whenever a device proactively sends a message to the IoT Agent, it should tranform its data to the appropriate NGSI format, and send it to the Context Broker as an `updateContext` request.

### Features
These are the features an IoT Agent is supposed to expose (those not supported yet by this library are marked as PENDING):
* **Device registration**: multiple devices will be connected to each IoT Agent, each one of those mapped to a CB entity. The IoT Agent will register itself as a Context Provider for each device, answering to requests and updates on any lazy attribute of the device.
* **Device information update**: whenever a device haves new measures to publish, it should send the information to the IoT Agent in its own native language. This message should , in turn, should be sent as an `updateContext` request to the Context Broker, were the measures will be updated in the device entity. 
* **Device command execution and value updates**: as a Context Provider, the IoT Agent should receive update operations from the Context Broker subscriptions, and relay them to the corresponding device (decoding it using its ID and Type, and other possible metadata). This commands will arrive as `updateContext` operations redirected from the Context Broker to the IoT Agent (Command execution PENDING; value updates available).
* **Device management**: the IoT Agent should offer a device repository where the devices can be registered, holding data needed for the connection to the Context Broker as the following: service and subservice for the device, API Key the device will be using to connect to the IoT Agent, Trust token the device will be using to retrieve the Keystone token to connect to the Context Broker.
* **Device provisioning**: the IoT Agent should offer an external API to make a preprovision of any devices. This preprovision should enable the user to customize the device`s entity name and type as well as their service information.
* **Type configuration**: if a device is registered without a preregistration, only its `id` and `type` attributes are mandatory. The IoT Agent should provide a mechanism to provide default values to the device attributes based on its type (a JSON configuration file with the default values per type).

Almost all of these features are common for every agent, so they can be abstracted into a library or external module. The objective of this project is to provide that abstraction. As all this common tasks are abstracted, the main task of the concrete IoT Agent implementations will be to map between the native device protocol and the library API.

The following figure offers a graphical example of how a COAP IoT Agent work, ordered from the registration of the device to a command update to the device.

![General ](https://raw.github.com/dmoranj/iotagent-node-lib/develop/img/iotAgentLib.png "Architecture Overview")

### Implementation decisions
Given the aforementioned requirements, there are some aspects of the implementation that were chosen, and are particularly under consideration:
* The IoT Agent Lib will save its configuration as a text file, and it will be updated using an external API (the API consisting in a single REST resource with a JSON object, that will be in the internal configuration format).
* Aside from its text configuration, the IoT Agent Lib is considered to be stateless. To be precise, the library mantains a state (the list of entities/devices whose information the agent can provide) but that state is considered to be transient. It's up to the particular implementation of the agent to consider whether it should have a persistent storage to hold the device information (so the internal list of devices is read from a DB) or to register the devices each time a device sends a measure. To this extent, two flavours of the Device Registry has been provided: a transient one (In-memory Registyr) and a persistent one (based in MongoDB).
* The IoT Agent does not care about the origin of the data, its type or structure. The mapping from raw data to the entity model, if there is any, is a responsability of the particular IoT Agent implementation, or of another third party library.

## <a name="usage"/> Usage
### Library usage
#### General review
In order to use the library, add the following dependency to your package.json file:
```
"iotagent-node-lib": "*"
```
In order to use this library, first you must require it:
```
var iotagentLib = require('iotagent-node-lib');
```
As a Lightweight M2M Server, the library supports four groups of features, one for each direction of the communication: client-to-server and server-to-client (and each flow both for the client and the server). Each feature set is defined in the following sections.

#### Operations
##### iotagentLib.activate
###### Signature
```
function activate(newConfig, callback)
```
###### Description
Activates the IoT Agent to start listening for NGSI Calls (acting as a Context Provider). It also creates the device registry for the IoT Agent (based on the deviceRegistry.type configuration option).
###### Params
* newConfig: Configuration of the Context Server (described in the [Configuration](#configuration) section).

##### iotagentLib.deactivate()
###### Signature
```
function deactivate(callback)
```
###### Description
Stops the HTTP server.
###### Params

##### iotagentLib.register()
###### Signature
```
registerDevice(deviceObj, callback)
```
###### Description
Register a new device in the IoT Agent. This registration will also trigger a Context Provider registration in the Context Broker for all its lazy attributes.

The device Object can have the following attributes:
* id: Device ID of the device.
* type: type to be assigned to the device.
* name: name that will be used for the Entity representing the device in the Context Broker.
* service: name of the service associated with the device.
* subservice: name of the subservice associated with th device.
* lazy:	list of lazy attributes with their types.
* active: list of active attributes with their types.
* staticAttributes: list of NGSI attributes to add to the device entity 'as is' in updates, queries and registrations.
* internalAttributes: optional section with free format, to allow specific IoT Agents to store information along with the devices in the Device Registry.


The device id and type are required fields for any registration. The rest of the attributes are optional, but, if they are not present in the function call arguments, the type must be registered in the configuration, so the service can infer their default values from the configured type. If an optional attribute is not given in the parameter list and there isn't a default configuration for the given type, a TypeNotFound error is raised.

If the device has been previously preprovisioned, the missing data will be completed with the values from the registered device.

###### Params
 * deviceObj: object containing all the information about the device to be registered (mandatory).
##### iotagentLib.unregister()
###### Signature
```
function unregisterDevice(id, callback)
```
###### Description
Unregister a device from the Context broker and the internal registry.
###### Params
 * id: Device ID of the device to register.
 
##### iotagentLib.update()
###### Signature
```
updateValue(deviceId, resource, apikey, attributes, deviceInformation, callback)
```
###### Description
Launches the updating process, getting the security token in case the authorization sequence is enabled. This method
can be invoked with an externally added deviceInformation object to overwrite the information on the configuration
(for preregistered devices).<commit_msg>MD fix, needed an extra line break in order to render correctly<commit_after># FIWARE IoT Agent Framework

## Index

* [Overview](#overview)
* [Usage](#usage)
* [IoT Library testing](#librarytesting)
* [Configuration](#configuration)
* [Device Provisioning API](#provisioningapi)
* [Configuration API](#configurationapi)
* [Secured access to the Context Broker](#securedaccess)
* [Development Documentation](#development)

## <a name="overview"/> Overview
### Description
This project aims to provide a node.js module to enable IoT Agent developers to build custom agents for their devices that can 
easily connect to NGSI Context Brokers (such as [Orion](https://github.com/telefonicaid/fiware-orion) ). 

An IoT Agent is a component that lets groups of devices send their data to and be managed from a FIWARE NGSI Context Broker using their own native protocols. IoT Agents should also be able to deal with security aspects of the fiware platform (authentication and authorization of the channel) and provide other common services to the device programmer.

### Device to NGSI Mapping
Each Device will be mapped as an Entity associated to a Context Provider: the Device Id will be mapped by default to the entity ID and the type of the entity will be selected by the IoT Agent in a protocol-dependent way (e.g: with different URLs for different types). Both the name and type will be configurable by the user, either by type configuration or with the device preprovisioning).

Each of the measures obtained from the device should be mapped to a different attribute. The name and type of the attribute will be configured by the user (globally for all the types in the IoT Agent configuration or in a per device basis preprovisioning the devices). Device measures can have two different behaviors:

* **Active attributes**: are measures that are pushed from the device to the IoT agent. This measure changes will be sent to the Context Broker as updateContext requests over the device entity. NGSI queries to the context broker will be resolved in the Broker database.
* **Lazy attributes**: some sensors will be passive, and will wait for the IoT Agent to request for data. For those measures, the IoT Agent will register itself in the Context Broker as a Context Provider (for all the lazy measures of that device), so if any component asks the Context Broker for the value of that sensor, its request will be redirected to the IoT Agent (that behaves as a NGSI10 Context Provider). 

The following sequence diagram shows the different NGSI interactions an IoT Agent makes with the Context Broker, explained in the following subsections (using the example of a OMA Lightweight M2M device).

![General ](https://raw.github.com/dmoranj/iotagent-node-lib/develop/img/ngsiInteractions.png "NGSI Interactions")

Be aware that the IoT Agents are only required to support NGSI10 operations `updateContext` and `queryContext` in their standard formats (both in XML and JSON) but will not answer to NGSI9 operations (or NGSI convenience operations of any kind).

#### Registration
Whenever a device is registered, the IoT Agent reads the device's entity information from the request or, it that information is not in the request, from the default values for that type of device. Among this information, theres should be the list of device attributes that will be considered lazy (or passive). With this information, the IoT Agent sends a new `registerContext` request to the Context Broker, registering itself as ContextProvider of all the lazy attributes for the device's entity. The `registrationId` is then stored along the other device information inside the IoT Agent device regitry.

As NGSI9 does not allow the context registrations to be removed, when the device is removed from the IoT Agent, the registration is updated to an expiration date of 1s, so it is effectively disabled. Once it has been disabled, the device is removed from the IoT Agent's internal registry.

#### Lazy attributes
When a request for data from a lazy attribute arrives to the Context Broker, it forwards the request to the Context Provider of that entity, in this case the IoT Agent. The IoT Agent will in turn ask the device for the information needed, transform that information to a NSGI format and return it to the Context Broker. The latter will the forward the response to the caller, transparently.

#### Commands
Commands are modelled as updates over a lazy attribute. As in the case of the lazy attributes, updates over a command will be forwarded by the Context Broker to the IoT Agent, that will in turn interact with the device to perform the requested action. Parameters for the command will be passed inside the command value.

There are two differences with the lazy attributes:
* First of all, for every command defined in a device, two new attributes are created in the entity with the same name as the command but with a prefix:
	* '_status': this attribute reflect the current execution status of the command. When a command request is issued by the Context Broker, the IoT Agent library generates this attribute with 'PENDING' value. The value of this attribute will be changed each time a command error or result is issued to the IoT Agent.
	* '_result': this attribute reflect the result of the execution of the defined command.

* Commands can also be updated when new information about its execution arrives to the agent. This information will be mapped to the command's utility attributes `_status` and `_result` leaving alone the command attribute itself. The values for this attributes are stored locally in the Context Broker (instead of being redirected with the Context Provider operations).

#### Active attributes
Whenever a device proactively sends a message to the IoT Agent, it should tranform its data to the appropriate NGSI format, and send it to the Context Broker as an `updateContext` request.

### Features
These are the features an IoT Agent is supposed to expose (those not supported yet by this library are marked as PENDING):
* **Device registration**: multiple devices will be connected to each IoT Agent, each one of those mapped to a CB entity. The IoT Agent will register itself as a Context Provider for each device, answering to requests and updates on any lazy attribute of the device.
* **Device information update**: whenever a device haves new measures to publish, it should send the information to the IoT Agent in its own native language. This message should , in turn, should be sent as an `updateContext` request to the Context Broker, were the measures will be updated in the device entity. 
* **Device command execution and value updates**: as a Context Provider, the IoT Agent should receive update operations from the Context Broker subscriptions, and relay them to the corresponding device (decoding it using its ID and Type, and other possible metadata). This commands will arrive as `updateContext` operations redirected from the Context Broker to the IoT Agent (Command execution PENDING; value updates available).
* **Device management**: the IoT Agent should offer a device repository where the devices can be registered, holding data needed for the connection to the Context Broker as the following: service and subservice for the device, API Key the device will be using to connect to the IoT Agent, Trust token the device will be using to retrieve the Keystone token to connect to the Context Broker.
* **Device provisioning**: the IoT Agent should offer an external API to make a preprovision of any devices. This preprovision should enable the user to customize the device`s entity name and type as well as their service information.
* **Type configuration**: if a device is registered without a preregistration, only its `id` and `type` attributes are mandatory. The IoT Agent should provide a mechanism to provide default values to the device attributes based on its type (a JSON configuration file with the default values per type).

Almost all of these features are common for every agent, so they can be abstracted into a library or external module. The objective of this project is to provide that abstraction. As all this common tasks are abstracted, the main task of the concrete IoT Agent implementations will be to map between the native device protocol and the library API.

The following figure offers a graphical example of how a COAP IoT Agent work, ordered from the registration of the device to a command update to the device.

![General ](https://raw.github.com/dmoranj/iotagent-node-lib/develop/img/iotAgentLib.png "Architecture Overview")

### Implementation decisions
Given the aforementioned requirements, there are some aspects of the implementation that were chosen, and are particularly under consideration:
* The IoT Agent Lib will save its configuration as a text file, and it will be updated using an external API (the API consisting in a single REST resource with a JSON object, that will be in the internal configuration format).
* Aside from its text configuration, the IoT Agent Lib is considered to be stateless. To be precise, the library mantains a state (the list of entities/devices whose information the agent can provide) but that state is considered to be transient. It's up to the particular implementation of the agent to consider whether it should have a persistent storage to hold the device information (so the internal list of devices is read from a DB) or to register the devices each time a device sends a measure. To this extent, two flavours of the Device Registry has been provided: a transient one (In-memory Registyr) and a persistent one (based in MongoDB).
* The IoT Agent does not care about the origin of the data, its type or structure. The mapping from raw data to the entity model, if there is any, is a responsability of the particular IoT Agent implementation, or of another third party library.

## <a name="usage"/> Usage
### Library usage
#### General review
In order to use the library, add the following dependency to your package.json file:
```
"iotagent-node-lib": "*"
```
In order to use this library, first you must require it:
```
var iotagentLib = require('iotagent-node-lib');
```
As a Lightweight M2M Server, the library supports four groups of features, one for each direction of the communication: client-to-server and server-to-client (and each flow both for the client and the server). Each feature set is defined in the following sections.

#### Operations
##### iotagentLib.activate
###### Signature
```
function activate(newConfig, callback)
```
###### Description
Activates the IoT Agent to start listening for NGSI Calls (acting as a Context Provider). It also creates the device registry for the IoT Agent (based on the deviceRegistry.type configuration option).
###### Params
* newConfig: Configuration of the Context Server (described in the [Configuration](#configuration) section).

##### iotagentLib.deactivate()
###### Signature
```
function deactivate(callback)
```
###### Description
Stops the HTTP server.
###### Params

##### iotagentLib.register()
###### Signature
```
registerDevice(deviceObj, callback)
```
###### Description
Register a new device in the IoT Agent. This registration will also trigger a Context Provider registration in the Context Broker for all its lazy attributes.

The device Object can have the following attributes:
* id: Device ID of the device.
* type: type to be assigned to the device.
* name: name that will be used for the Entity representing the device in the Context Broker.
* service: name of the service associated with the device.
* subservice: name of the subservice associated with th device.
* lazy:	list of lazy attributes with their types.
* active: list of active attributes with their types.
* staticAttributes: list of NGSI attributes to add to the device entity 'as is' in updates, queries and registrations.
* internalAttributes: optional section with free format, to allow specific IoT Agents to store information along with the devices in the Device Registry.


The device id and type are required fields for any registration. The rest of the attributes are optional, but, if they are not present in the function call arguments, the type must be registered in the configuration, so the service can infer their default values from the configured type. If an optional attribute is not given in the parameter list and there isn't a default configuration for the given type, a TypeNotFound error is raised.

If the device has been previously preprovisioned, the missing data will be completed with the values from the registered device.

###### Params
 * deviceObj: object containing all the information about the device to be registered (mandatory).

##### iotagentLib.unregister()
###### Signature
```
function unregisterDevice(id, callback)
```
###### Description
Unregister a device from the Context broker and the internal registry.
###### Params
 * id: Device ID of the device to register.
 
##### iotagentLib.update()
###### Signature
```
updateValue(deviceId, resource, apikey, attributes, deviceInformation, callback)
```
###### Description
Launches the updating process, getting the security token in case the authorization sequence is enabled. This method
can be invoked with an externally added deviceInformation object to overwrite the information on the configuration
(for preregistered devices).
{'fraction_non_alphanumeric': 0.05296206316576341, 'fraction_numerical': 0.0006727714445897963, 'mean_word_length': 4.897288957460877, 'pattern_counts': {'":': 2, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4056961', 'n_tokens_mistral': 6180, 'n_tokens_neox': 5811, 'n_words': 4014}