<commit_before>#<a name="top"></a>Performance tuning

* [MongoDB version](#mongodb-version)
* [Database indexes](#database-indexes)
* [Write concern](#write-concern)
* [Notification modes and performance](#notification-modes-and-performance)
* [HTTP server tuning](#http-server-tuning)
* [Thread pool considerations](#thread-pool-considerations)
* [Identifying bootlenecks looking at semWait statistics](#identifying-bootlenecks-looking-at-semwait-statistics)
* [Log impact in performance](#log-impact-in-performance)
* [Mutex policy impact in performance](#mutex-policy-impact-in-performance)
* [Outgoing HTTP connections timeout](#outgoing-http-connetions-timeout)
* [Subscription cache](#subscription-cache)

##  MongoDB version

Since version 0.21.0, Orion supports both MongoDB 2.6 or MongoDB 3.0 without differences from a functional
point of view. However, MongoDB 2.6 implements a per-collection lock, while MongoDB 3.0 (when configured
to use WireTiger storage engine) implements per-document lock. Thus, the lock system in MongoDB3.0
(with WireTiger) is less restrictive than the one used by MongoDB 2.6.

In sum, from a performance point of view, it is recommended to use MongoDB 3.0 with WireTiger, especially
in update-intensive scenarios.

[Top](#top)

## Database indexes

Orion Context Broker doesn't create any index in any database collection except for one exception,
described at the end of this section) in order to let flexibility to database administrators. Take
into account that index usage involves a tradeoff between read efficiency (usage of indexes generally
speeds up reads) and write efficiency (the usage of indexes slow down writes) and storage (indexes
consume space in database and mapped RAM memory) and is the administrator (not Orion) who has
to decide what to priorize.

However, in order to help administrator in that task, the following indexes are recommended:

* Collection [entities](database_model.md#entities-collection)
  * `_id.id`
  * `_id.type`
  * `_id.servicePath`
  * `attrNames`
  * `creDate`

The only index that Orion Context Broker actually ensures is the "2dsphere" one in the `location.coords`
field in the entities collection, due to functional needs [geo-location functionality](../user/geolocation.md)to work.
The index is ensured at Orion startup or when entities are created for the first time.

You can find an analysis about indexes effect in [this document](https://github.com/telefonicaid/fiware-orion/blob/develop/doc/manuals/admin/extra/indexes_analysis.md), although
it is based in an old Orion version, so probably outdated.

[Top](#top)

## Write concern

[Write concern](https://docs.mongodb.org/manual/core/write-concern/) is a parameter for MongoDB write
operations. By default, Orion uses "acknowledgue" write concern that means that the Orion waits until
MongoDB confirm it has applyed the operation in memory. You can change this behavior with the
[`-writeConcern` CLI option](cli.md). When "unacknowledgue" write concern is configured, Orion doesn't
wait to get confirmation, so it can execute operation involving wring much faster.

Note, however, that there is a tradeoff between performance and reliability. Using "unacknowledgue" write
concern you will get better performance, but the risk to lost information (as Orion doesn't get any
indication regarding if the write operation were fine or not) is higher.

[Top](#top)

## Notification modes and performance

Orion can use different notifications modes, depending on the [`-notificationMode`](cli.md) value.

Default mode is transient. In this mode, each time a notification is sent, a new thread is created to deal
with it. Once the notification is sent and the response is received, the thread and the connection context
are destroyed. This is the recommended mode for low load scenarios. In high level cases it could lead to
a [thread exhaustion problem](#thread-pool-considerations).

Permanent mode is similar, except in that the connection context is not destroyed at the end. Thus,
new notifications associated to the same connection context (i.e. the same destination URL) could
reuse the connection and save the HTTP connection time (i.e. TCP handshake, etc.). Of course,
this requires that the server (i.e. the component received the notification) maintains also the connection
opened. Note that while in some cases the permanent mode could improve performance (as it saves
the time required to create and destroy HTTP connections), but in others it make cause that notifications
associated to the same connection context have to wait (as only one at the same time may use the
connection). In other words, only one notification thread can use the connection at a time, so if the
notification request/response transmission time is longer than notification inter-triggering time
then threads will block- You can detect this situation when the `connectionContext` value
[in statistics](statistics.md#semwait-block) is abnormally high.

Finally, threadpool mode is based on a queue for notifications and a pool of worker threads that
take notifications from the queue and actually send them on the wire, as shown in the figure below.
This is the recommended mode for high load scenarios, after a careful tuning on the queue length
and workers number. A good starting point is to set the number of workers to the number of expected
concurrent clients sending updates and queue limit as N times the number of workers (e.g. N equal
to 10, although it could be more or less depending on the expected update burst length). The statistics
on [the `notifQueue` block](statistics.md#notifqueue-block) could help you to tune.

![](notif_queue.png "notif_queue.png")

[Top](#top)

## HTTP server tuning

The REST API which Orion implements is provided by an HTTP server listening on port 1026 by default
(this can be overridden by the [`-port` CLI parameter](cli.md)). You can tune its behavior using the
following CLI parameters (see details at the corresponding document):

* **connectionMemory**. Sets the size of the connection memory buffer (in Kb) per connection used
  internally by the HTTP server library. Default value is 64 Kb.

* **maxConnections**. Maximum number of simultaneous connections. Default value is "unlimited" (limited
   by max file descriptors of operating system).

* **reqPoolSize**. Size of thread pool for incoming connections. Default value is 0, meaning no thread pool,
  i.e. a new thread is created to manage each new incoming HTTP request and destroyed after use.

Given that thread creation and destruction is a costly operation, it is recommend to use `-reqPoolSize` in
high load scenarios. The other two parameters use to work well with the default values.

![](requests_queue.png "requests_queue.png")

[Top](#top)

## Thread pool considerations

Orion can use thread pools at two different points: to process incoming HTTP requests at the API endpoint
(managed by `-reqPoolSize`) and to process outgoing notifications (using the threadpool notification mode).

There is a tradeoff between using or not using thread pools. On the one side, using thread pools is
beneficial as it saves thread creation/destruction times. In addition, they make the behavior of Orion more
predictable, taking into account that operating system use to stablish a thread limit per process and
threadpools is a way of guarantee that the Orion process doesn't goes beyond that limit (typically, you
can change the limit with the ulimit command).

On the other hand, setting thread pools is a way of "capping" throughput. If the thread workers are busy
all the time, at the end the queue saturates and you will end losing incoming request or ongoing
notifications.

Finally, you may have thread exhaustion problems if don’t use threadpools. You can detect that
situation by two symptoms. First, an unexpectedly high number of thread associated to the process.<commit_msg>FIX typos in doc<commit_after>#<a name="top"></a>Performance tuning

* [MongoDB version](#mongodb-version)
* [Database indexes](#database-indexes)
* [Write concern](#write-concern)
* [Notification modes and performance](#notification-modes-and-performance)
* [HTTP server tuning](#http-server-tuning)
* [Thread pool considerations](#thread-pool-considerations)
* [Identifying bootlenecks looking at semWait statistics](#identifying-bootlenecks-looking-at-semwait-statistics)
* [Log impact in performance](#log-impact-in-performance)
* [Mutex policy impact in performance](#mutex-policy-impact-in-performance)
* [Outgoing HTTP connections timeout](#outgoing-http-connections-timeout)
* [Subscription cache](#subscription-cache)

##  MongoDB version

Since version 0.21.0, Orion supports both MongoDB 2.6 or MongoDB 3.0 without differences from a functional
point of view. However, MongoDB 2.6 implements a per-collection lock, while MongoDB 3.0 (when configured
to use WireTiger storage engine) implements per-document lock. Thus, the lock system in MongoDB3.0
(with WireTiger) is less restrictive than the one used by MongoDB 2.6.

In sum, from a performance point of view, it is recommended to use MongoDB 3.0 with WireTiger, especially
in update-intensive scenarios.

[Top](#top)

## Database indexes

Orion Context Broker doesn't create any index in any database collection except for one exception,
described at the end of this section) in order to let flexibility to database administrators. Take
into account that index usage involves a tradeoff between read efficiency (usage of indexes generally
speeds up reads) and write efficiency (the usage of indexes slow down writes) and storage (indexes
consume space in database and mapped RAM memory) and is the administrator (not Orion) who has
to decide what to priorize.

However, in order to help administrator in that task, the following indexes are recommended:

* Collection [entities](database_model.md#entities-collection)
  * `_id.id`
  * `_id.type`
  * `_id.servicePath`
  * `attrNames`
  * `creDate`

The only index that Orion Context Broker actually ensures is the "2dsphere" one in the `location.coords`
field in the entities collection, due to functional needs [geo-location functionality](../user/geolocation.md)to work.
The index is ensured at Orion startup or when entities are created for the first time.

You can find an analysis about indexes effect in [this document](https://github.com/telefonicaid/fiware-orion/blob/develop/doc/manuals/admin/extra/indexes_analysis.md), although
it is based in an old Orion version, so probably outdated.

[Top](#top)

## Write concern

[Write concern](https://docs.mongodb.org/manual/core/write-concern/) is a parameter for MongoDB write
operations. By default, Orion uses "acknowledgue" write concern that means that the Orion waits until
MongoDB confirm it has applyed the operation in memory. You can change this behavior with the
[`-writeConcern` CLI option](cli.md). When "unacknowledgue" write concern is configured, Orion doesn't
wait to get confirmation, so it can execute operation involving wring much faster.

Note, however, that there is a tradeoff between performance and reliability. Using "unacknowledgue" write
concern you will get better performance, but the risk to lost information (as Orion doesn't get any
indication regarding if the write operation were fine or not) is higher.

[Top](#top)

## Notification modes and performance

Orion can use different notifications modes, depending on the [`-notificationMode`](cli.md) value.

Default mode is transient. In this mode, each time a notification is sent, a new thread is created to deal
with it. Once the notification is sent and the response is received, the thread and the connection context
are destroyed. This is the recommended mode for low load scenarios. In high level cases it could lead to
a [thread exhaustion problem](#thread-pool-considerations).

Permanent mode is similar, except in that the connection context is not destroyed at the end. Thus,
new notifications associated to the same connection context (i.e. the same destination URL) could
reuse the connection and save the HTTP connection time (i.e. TCP handshake, etc.). Of course,
this requires that the server (i.e. the component received the notification) maintains also the connection
opened. Note that while in some cases the permanent mode could improve performance (as it saves
the time required to create and destroy HTTP connections), but in others it make cause that notifications
associated to the same connection context have to wait (as only one at the same time may use the
connection). In other words, only one notification thread can use the connection at a time, so if the
notification request/response transmission time is longer than notification inter-triggering time
then threads will block- You can detect this situation when the `connectionContext` value
[in statistics](statistics.md#semwait-block) is abnormally high.

Finally, threadpool mode is based on a queue for notifications and a pool of worker threads that
take notifications from the queue and actually send them on the wire, as shown in the figure below.
This is the recommended mode for high load scenarios, after a careful tuning on the queue length
and workers number. A good starting point is to set the number of workers to the number of expected
concurrent clients sending updates and queue limit as N times the number of workers (e.g. N equal
to 10, although it could be more or less depending on the expected update burst length). The statistics
on [the `notifQueue` block](statistics.md#notifqueue-block) could help you to tune.

![](notif_queue.png "notif_queue.png")

[Top](#top)

## HTTP server tuning

The REST API which Orion implements is provided by an HTTP server listening on port 1026 by default
(this can be overridden by the [`-port` CLI parameter](cli.md)). You can tune its behavior using the
following CLI parameters (see details at the corresponding document):

* **connectionMemory**. Sets the size of the connection memory buffer (in Kb) per connection used
  internally by the HTTP server library. Default value is 64 Kb.

* **maxConnections**. Maximum number of simultaneous connections. Default value is "unlimited" (limited
   by max file descriptors of operating system).

* **reqPoolSize**. Size of thread pool for incoming connections. Default value is 0, meaning no thread pool,
  i.e. a new thread is created to manage each new incoming HTTP request and destroyed after use.

Given that thread creation and destruction are costly operations, it is recommend to use `-reqPoolSize` in
high load scenarios. The other two parameters use to work well with the default values.

![](requests_queue.png "requests_queue.png")

[Top](#top)

## Thread pool considerations

Orion can use thread pools at two different points: to process incoming HTTP requests at the API endpoint
(managed by `-reqPoolSize`) and to process outgoing notifications (using the threadpool notification mode).

There is a tradeoff between using or not using thread pools. On the one side, using thread pools is
beneficial as it saves thread creation/destruction times. In addition, they make the behavior of Orion more
predictable, taking into account that operating system use to stablish a thread limit per process and
threadpools is a way of guarantee that the Orion process doesn't goes beyond that limit (typically, you
can change the limit with the ulimit command).

On the other hand, setting thread pools is a way of "capping" throughput. If the thread workers are busy
all the time, at the end the queue saturates and you will end losing incoming request or ongoing
notifications.

Finally, you may have thread exhaustion problems if don’t use threadpools. You can detect that
situation by two symptoms. First, an unexpectedly high number of thread associated to the process.
{'fraction_non_alphanumeric': 0.06122191365816713, 'fraction_numerical': 0.0035344609946983086, 'mean_word_length': 4.954528372792184, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3691271', 'n_tokens_mistral': 4080, 'n_tokens_neox': 3834, 'n_words': 2185}