<commit_before>Testing Modules on Simulators
================================

.. warning:: *The section is under development!*

Stream Juggler provides a user with a range of simulators for module testing purposes. A simulator is a ready-to-use environment that allows you to rapidly test modules during the development process.

Four types of simulators is provided - one per each module type. Choose that you want to use in accordance with the type of a module you want to test. And follow the instructions below.

.. _Input_Engine_Simulator:

Input Engine Simulator
-----------------------------

.. warning:: *The section is under development!*

It is a class for testing an implementation of an :ref:`input_module` (Executor).

Simulator imitates the behavior of the :ref:`Input_Streaming_Engine`: it sends byte buffer to Executor, gets input envelopes from it, checks envelopes on duplicate (if it is necessary), and builds :ref:`Input_Engine_Simulator_Output_Data`.

Constructor arguments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. csv-table:: 
 :header: "Argument", "Type", "Description"
 :widths: 25, 25, 50  

 "executor*", "InputStreamingExecutor[T]", "Implementation of :ref:`input_module` under testing"
 "evictionPolicy*", "InputInstanceEvictionPolicy", "A field of an instance (:ref:`REST_API_Instance_Create`)"
 "separator", "String", "Delimeter between data records (empty string by default)"
 "charset", "Charset", "Encoding of incoming data (UTF-8 by default)"

.. note:: `*` - required field
.. important:: T - is a type of data created by Executor 

The data record is a string that will be parsed by ``executor.parse()`` to some entity.

The simulator provides the following methods:

* ``prepare(record: String)`` - writes one data record to a byte buffer.
* ``prepare(records: Seq[String])`` - writes a collection of data records to a byte buffer.
* ``process(duplicateCheck: Boolean, clearBuffer: Boolean = true): Seq[OutputData[T]]`` - sends byte buffer to the executor as long as it can tokenize the buffer. The ``duplicateCheck`` argument indicates that every envelope has to be checked on duplication, the ``clearBuffer`` argument indicates that byte buffer with data records has to be cleared after processing ('true' by default). Method returns list of :ref:`Input_Engine_Simulator_Output_Data`.
* ``clear()`` - clears a byte buffer with data records.

.. _Input_Engine_Simulator_Output_Data:

Output Data
~~~~~~~~~~~~~~~~~

Provides information on the processing of incoming data by the :ref:`input_module`.<commit_msg>Update SJ_Simulators.rst<commit_after>Testing Modules on Simulators
================================

.. warning:: *The section is under development!*

Stream Juggler provides a user with a range of simulators for module testing purposes. A simulator is a ready-to-use environment that allows you to rapidly test modules during the development process.

Four types of simulators is provided - one per each module type. Choose that you want to use in accordance with the type of a module you want to test. And follow the instructions below.

.. _Input_Engine_Simulator:

Input Engine Simulator
-----------------------------

.. warning:: *The section is under development!*

It is a class for testing an implementation of an :ref:`input_module` (Executor).

Simulator imitates the behavior of the :ref:`Input_Streaming_Engine`: it sends byte buffer to Executor, gets input envelopes from it, checks envelopes on duplicate (if it is necessary), and builds :ref:`Input_Engine_Simulator_Output_Data`.

Constructor arguments
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. csv-table:: 
 :header: "Argument", "Type", "Description"
 :widths: 25, 25, 50  

 "executor*", "InputStreamingExecutor[T]", "Implementation of :ref:`input_module` under testing"
 "evictionPolicy*", "InputInstanceEvictionPolicy", "A field of an instance (:ref:`REST_API_Instance_Create`)"
 "separator", "String", "Delimeter between data records (empty string by default)"
 "charset", "Charset", "Encoding of incoming data (UTF-8 by default)"

.. note:: `*` - required field
.. important:: T - is a type of data created by Executor 

The data record is a string that will be parsed by ``executor.parse()`` to some entity.

Simulator provides the following methods:

* ``prepare(record: String)`` - writes one data record to a byte buffer.
* ``prepare(records: Seq[String])`` - writes a collection of data records to a byte buffer.
* ``process(duplicateCheck: Boolean, clearBuffer: Boolean = true): Seq[OutputData[T]]`` - sends byte buffer to the executor as long as it can tokenize the buffer. The ``duplicateCheck`` argument indicates that every envelope has to be checked on duplication, the ``clearBuffer`` argument indicates that byte buffer with data records has to be cleared after processing ('true' by default). Method returns list of :ref:`Input_Engine_Simulator_Output_Data`.
* ``clear()`` - clears a byte buffer with data records.

.. _Input_Engine_Simulator_Output_Data:

Output Data
~~~~~~~~~~~~~~~~~

Provides information on the processing of incoming data by the :ref:`input_module`.
{'fraction_non_alphanumeric': 0.12855740922473013, 'fraction_numerical': 0.002747791952894995, 'mean_word_length': 4.967213114754099, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1339728', 'n_tokens_mistral': 1331, 'n_tokens_neox': 1275, 'n_words': 611}