<commit_before>Persistent Message Queues Overview
==================================
<NAME>
v1.0, 2016-05:
	Initial draft
:doctype: book

Persistent queues
-----------------
EnduroX system is built on kernel based real-time IPC queues. Which by definition are not persistent.
When server is restarted, memory data is lost, thus the data in message queues are lost. In some of
the application scenarios persistent messaging is required. Thus EnduroX offers standard ATMI calls
such as 'tpenqueue()' and 'tpdequeue()'. System also supports automatic queues, where messages
are automatically forwarded to configured services.

Messages are stored in file system. To ensure transactional operation, file move/rename is
employed which by it self (in the terms of one file system) is transactional.

The queues are grouped into queue spaces. Queue space grouping element of multiple queue servers.
Queue space basically is service name for backing 'tmqueue' server. Each queue server
can stores it's data in specified file system folders. The 'tmqueue' server works in tandem with
'tmsrv' to ensure the compatibility with global transaction processing.

How the persistent queues work?
-------------------------------
The persistent queues in EnduroX application server are provided by special ATMI server, named
"tmqueue". To start using these queus, firstly you have to configure the 'tmqueue' and paired
'tmsrv'. 

The queue processing can be devided in following steps:

1. 'tmqueue' advertizes queue space services
2. Caller invokes 'tpenqueue()' ATMI API call. This calls the 'tmqueue' server with passed in buffer
data and other flags.
3. 'tmqueue' server recieves the request, saves lookups the queue, and creates the linked list in
memory where to store the message. In the same time message is written to disk in 'active' folder.
Initially message is marked as locked, the message becomes dequeueable at the point when XA 
transaction is committed. At the commit point, the 'tmsrv' with loaded EnduroX queue driver,
completes the message (moves it to 'committed' folder). At the smae point 'tmsrv' sends the notification
to 'tmqueue' to unlock the message. NOTE: It is not possible to enqueue and dequeue same message
in single transaction.
4. When message is unlocked, it is available for 'tpdequeue()'. Where the application invokes this function
and, it calls the 'tmqueue()' for the message. If message is found it again becomes locked, and command
file is issued to disk for message file removal. Once the dequeue transaction is committed, the XA driver
completes the operation, by removing the command file, message file and sending notification back
to 'tmqueue', that command 'REMOVE' is completed. At this point 'tmqueue' server removes the message
completely from memory.
5. In case if queue is defined as 'automatic', the 'forward' threads from 'tmqueue' server
begins sending the message to destination ATMI service. If service call fails, the call is 
retried configured number of times. If call succeeds and reply queue is sent, then message is submitted
to reply queue. If message fails (number of attempts exceeded) and failure queue is defined,
the forwarder thread will submit the message to failure queue. 'TPQCTL' flags in this scenario is
preserved.

Schematically internals of the 'tmqueue' and API works can be displayed as follows:

image:tmqinternals.png[caption="Figure 1: ", title="tmqueue internals", alt="tmqueue"]


How to use EnduroX persistent queues
------------------------------------
This section gives some oversight how to use persistent queues. There are two type 
of usages possible. One is that process submits the message to the queue and another
process manually dequeues the message, that is depicted here:

[dia, tpenqueue_tpdequeue.dia, tpenqueue_tpdequeue.png, x300]
-------------------------------
-------------------------------

And another use is that process does 'tpenqueue()' to automatic queue, and message is automatically
forwarded to destination service. The schematics looks like this:

[dia, tpenqueue_auto.dia, tpenqueue_auto.png, x300]
-------------------------------
-------------------------------

For 'tpenqueue()' and 'tpdequeue()' passed in buffers must be allocated with 'tpalloc()'.
For UBF, STRING and JSON buffers, the actual buffer length on enqueue doesn't matter, it is
detected from data data inside. For array buffer, it does play a role. When doing
'tpdequeue()', the the buffer type might be changed, if message have a different data
type.


'tmqueue' ATMI server configuration
-----------------------------------
To configure queue sub-system, you need to start at-least one instance of 'tmqueue' server and
one instance of 'tmsrv'. They both must run in the same XA environment. For running just pure
ATMI client, following 'ndrxconfig.xml' shall be done:

---------------------------------------------------------------------
...
        <servers>
            <server name="tmsrv">
                <max>1</max>
                <srvid>1</srvid>
                <sysopt>-e /opt/app1/log/tmsrv.log -r -- -t1 -l/opt/app1/var/RM1</sysopt>
            </server>
            <server name="tmqueue">
                <max>1</max>
                <srvid>100</srvid>
                <sysopt>-e /opt/app1/log/tmqueue.log -r -- -m MYSPACE -q /opt/app1/conf/q.conf -s1</sysopt>
            </server>
        </servers>
...
---------------------------------------------------------------------



Queue configuration
-------------------
We will configure three queues here. The default queue (recommended), one manual queue
and one automatic queue.


Sample ATMI client for enqueue & dequeue
----------------------------------------
This section contains C code for ATMI client which enqueues the message to Q and the 
dequeues it.


Managing the runtime<commit_msg>Works on documentation<commit_after>Persistent Message Queues Overview
==================================
<NAME>
v1.0, 2016-05:
	Initial draft
:doctype: book

Persistent queues
-----------------
EnduroX system is built on kernel based real-time IPC queues. Which by definition are not persistent.
When server is restarted, memory data is lost, thus the data in message queues are lost. In some of
the application scenarios persistent messaging is required. Thus EnduroX offers standard ATMI calls
such as 'tpenqueue()' and 'tpdequeue()'. System also supports automatic queues, where messages
are automatically forwarded to configured services.

Messages are stored in file system. To ensure transactional operation, file move/rename is
employed which by it self (in the terms of one file system) is transactional.

The queues are grouped into queue spaces. Queue space grouping element of multiple queue servers.
Queue space basically is service name for backing 'tmqueue' server. Each queue server
can stores it's data in specified file system folders. The 'tmqueue' server works in tandem with
'tmsrv' to ensure the compatibility with global transaction processing.

How the persistent queues work?
-------------------------------
The persistent queues in EnduroX application server are provided by special ATMI server, named
"tmqueue". To start using these queus, firstly you have to configure the 'tmqueue' and paired
'tmsrv'. 

The queue processing can be devided in following steps:

1. 'tmqueue' advertizes queue space services
2. Caller invokes 'tpenqueue()' ATMI API call. This calls the 'tmqueue' server with passed in buffer
data and other flags.
3. 'tmqueue' server recieves the request, saves lookups the queue, and creates the linked list in
memory where to store the message. In the same time message is written to disk in 'active' folder.
Initially message is marked as locked, the message becomes dequeueable at the point when XA 
transaction is committed. At the commit point, the 'tmsrv' with loaded EnduroX queue driver,
completes the message (moves it to 'committed' folder). At the smae point 'tmsrv' sends the notification
to 'tmqueue' to unlock the message. NOTE: It is not possible to enqueue and dequeue same message
in single transaction.
4. When message is unlocked, it is available for 'tpdequeue()'. Where the application invokes this function
and, it calls the 'tmqueue()' for the message. If message is found it again becomes locked, and command
file is issued to disk for message file removal. Once the dequeue transaction is committed, the XA driver
completes the operation, by removing the command file, message file and sending notification back
to 'tmqueue', that command 'REMOVE' is completed. At this point 'tmqueue' server removes the message
completely from memory.
5. In case if queue is defined as 'automatic', the 'forward' threads from 'tmqueue' server
begins sending the message to destination ATMI service. If service call fails, the call is 
retried configured number of times. If call succeeds and reply queue is sent, then message is submitted
to reply queue. If message fails (number of attempts exceeded) and failure queue is defined,
the forwarder thread will submit the message to failure queue. 'TPQCTL' flags in this scenario is
preserved.

Schematically internals of the 'tmqueue' and API works can be displayed as follows:

image:tmqinternals.png[caption="Figure 1: ", title="tmqueue internals", alt="tmqueue"]


How to use EnduroX persistent queues
------------------------------------
This section gives some oversight how to use persistent queues. There are two type 
of usages possible. One is that process submits the message to the queue and another
process manually dequeues the message, that is depicted here:

[dia, tpenqueue_tpdequeue.dia, tpenqueue_tpdequeue.png, x300]
-------------------------------
-------------------------------

And another use is that process does 'tpenqueue()' to automatic queue, and message is automatically
forwarded to destination service. The schematics looks like this:

[dia, tpenqueue_auto.dia, tpenqueue_auto.png, x300]
-------------------------------
-------------------------------

For 'tpenqueue()' and 'tpdequeue()' passed in buffers must be allocated with 'tpalloc()'.
For UBF, STRING and JSON buffers, the actual buffer length on enqueue doesn't matter, it is
detected from data data inside. For array buffer, it does play a role. When doing
'tpdequeue()', the the buffer type might be changed, if message have a different data
type.


'tmqueue' ATMI server configuration
-----------------------------------
To configure queue sub-system, you need to start at-least one instance of 'tmqueue' server and
one instance of 'tmsrv'. They both must run in the same XA environment. For running just pure
ATMI client, following entries to 'ndrxconfig.xml' shall be done (serverid and count (min/max) 
can be changed):

---------------------------------------------------------------------
...
        <servers>
            <server name="tmsrv">
                <max>1</max>
                <srvid>1</srvid>
                <sysopt>-e /opt/app1/log/tmsrv.log -r -- -t1 -l/opt/app1/var/RM1</sysopt>
            </server>
            <server name="tmqueue">
                <max>1</max>
                <srvid>100</srvid>
                <sysopt>-e /opt/app1/log/tmqueue.log -r -- -m MYSPACE -q /opt/app1/conf/q.conf -s1</sysopt>
            </server>
        </servers>
...
---------------------------------------------------------------------

From above example it could be seen, that there are no setup for folder where to store
the queue data. Queue folder is setup in in XA open string, thus it goes to
'NDRX_XA_OPEN_STR' and 'NDRX_XA_CLOSE' environment variables.

For example if we are going to store the message data into '/opt/app1/var/MYSPACE' folder, then
XA config looks like this:

---------------------------------------------------------------------
export NDRX_XA_RES_ID=1
export NDRX_XA_OPEN_STR="/opt/app1/var/MYSPACE"
export NDRX_XA_CLOSE_STR=$NDRX_XA_OPEN_STR
# Static registration:
export NDRX_XA_DRIVERLIB=libndrxxaqdisks.so
export NDRX_XA_RMLIB=libndrxxaqdisk.so
export NDRX_XA_LAZY_INIT=1

---------------------------------------------------------------------

It will use static registration XA driver.

Queue configuration
-------------------
We will configure three queues here. The default queue (recommended), one manual queue
and one automatic queue.


Sample ATMI client for enqueue & dequeue
----------------------------------------
This section contains C code for ATMI client which enqueues the message to Q and the 
dequeues it.


Managing the runtime
{'fraction_non_alphanumeric': 0.14221831829433842, 'fraction_numerical': 0.005589714924538848, 'mean_word_length': 2.820622330689445, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4031649', 'n_tokens_mistral': 3231, 'n_tokens_neox': 3102, 'n_words': 1603}