<commit_before>Scripts and their functions
===========================

Simple Producer
---------------
This is a command line script. It loads static text from a file (``ebu-tt-live-toolkit/ebu_tt_live/examples/simple_producer.txt``) and breaks it into a sequence of EBU-TT Live documents using natural language processing. Run it by entering the command ``ebu-simple-producer`` without any arguments.

The default carriage mechanism is WebSocket, so you will need to listen to ``ws://127.0.0.1:9000``. Conveniently, we've created an HTML page that does just that. After you launch ``ebu-simple-producer``, open ``test.html`` in your browser. The 'Broadcast message' field should be populated with the correct address (``ws://localhost:9000``). Click 'Connect' and then 'Subscribe'. You can also change the identifier for the sequence. The documents should appear in the window below.

Alternatively, the Simple Producer can use the file system as the carriage mechanism. Start it with the folder argument, like this: ``ebu-simple-producer --folder-export myFolder``. This saves the documents in ``myFolder`` together with a manifest file that can be used by the Simple Consumer (below). See `<filesystem_carriage_mechanism.html>`__ for more details about the file system carriage mechanism.

Simple consumer
---------------
This script reads and validates documents in a sequence. It performs both semantic and syntactic validation of the XML. By default, it uses WebSocket and listens to ``ws://localhost:9000``, but you can also point it to the file system. If you saved the documents in a folder (using the folder export argument of the Simple Producer), you can start it with the location of the manifest file, like so: ``ebu-simple-consumer --manifest-path myFolder/manifest_sequence_identifier.txt``.

User Input Producer
-------------------
This is a web page that adds a user interface and various configurations to the Simple Producer. First, start a consumer or a handover node node (``ebu-user-input-consumer``, ``ebu-user-input-forwarder`` or a handover node from the command line. Then, in your browser, open ``ebu_tt_live/ui/user_input_producer/user_input_producer.html`` and click 'Connect'. Select the sending mode (manual, scheduled or asynchronous). You should see the documents arriving in the command line window where ``ebu-user-input-consumer`` is listening. See detailed instructions here: :doc:`user_input_producer`.

User Input Consumer
-------------------
This is very similar to the Simple Consumer. It also performs validation but it does not have a manifest option. It provides a WebSocket connection point for the User Input Producer. You'll need to start either this or the User Input Forwarder before connecting the User Input Producer (but not both).

User Input Forwarder
--------------------
This script mimics a distribution node. It listens to documents coming from the User Input Producer on ``ws://127.0.0.1:9001`` and forwards them to any consumer listening on ``ws://127.0.0.1:9000``. Like the Simple Producer, it can also save the documents it receives to the file system. First, run it with the ``--folder-export`` argument like this: ``ebu-user-input-forwarder --folder-export myFolder``. Then launch the User Input Producer and connect. The sequence will be saved to ``myFolder`` along with the manifest file. The User Input Forwarder can also be used as an incoming connection point for WebSocket connections from sources other than the User Input Producer.

Handover Manager
----------------
This node implements the 'Who claimed control most recently' algorithm defined in the specification, with added functionality to allow messages to be broadcast between members of the authors group, which may be added to a future specification. This algorithm determines the output from multiple input sequences.
The Handover Manager is a specialised case of the switching node that bases its decisions on handover-related attributes in the document and its previous decisions.
There is no separate command to run this script. Start it with the ``ebu-run``, for example ``ebu-run --admin.conf=ebu_tt_live/examples/config/user_input_producer_handover.json`` for the default configuration. For detailed instruction on setting up the Handover Manager with the UIP see :doc:`user_input_producer`.

EBU-TT-D Encoder
----------------
This script is an extension of simple consumer and is responsible for resegmenting and converting the incoming
EBU-TT Live documents into EBU-TT-D documents that can be later used to be embedded in video streams such as DASH.
The script launches with the ``ebu-ebuttd-encoder`` command. There are switches to control the mediatime conversion
reference point and the segmentation interval. Please run ``ebu-ebuttd-encoder --help`` to find our more.

The ebu-run script
------------------
This script is capable of picking up a structured configuration file and use that to create the nodes and carriage mechanism objects that we would like to use and they can be even wired in the same configuration file together so in the end a complex system can be modeled using a single json file. Please refer to :py:mod:`ebu_tt_live.scripts.ebu_run` as well as :py:mod:`ebu_tt_live.config` to learn more about the way the configuration logic works. To get help on permitted options for the specified node(s) run ``ebu-run`` with a ``--help``. See :doc:`configurator`.<commit_msg>Reflow to avoid long lines<commit_after>Scripts and their functions
===========================

Simple Producer
---------------
This is a command line script. It loads static text from a file
(``ebu-tt-live-toolkit/ebu_tt_live/examples/simple_producer.txt``) and breaks it
into a sequence of EBU-TT Live documents using natural language processing. Run
it by entering the command ``ebu-simple-producer`` without any arguments.

The default carriage mechanism is WebSocket, so you will need to listen to
``ws://127.0.0.1:9000``. Conveniently, we've created an HTML page that does just
that. After you launch ``ebu-simple-producer``, open ``test.html`` in your
browser. The 'Broadcast message' field should be populated with the correct
address (``ws://localhost:9000``). Click 'Connect' and then 'Subscribe'. You can
also change the identifier for the sequence. The documents should appear in the
window below.

Alternatively, the Simple Producer can use the file system as the carriage
mechanism. Start it with the folder argument, like this: ``ebu-simple-producer
--folder-export myFolder``. This saves the documents in ``myFolder`` together
with a manifest file that can be used by the Simple Consumer (below). See
`<filesystem_carriage_mechanism.html>`__ for more details about the file system
carriage mechanism.

Simple consumer
---------------
This script reads and validates documents in a sequence. It performs both
semantic and syntactic validation of the XML. By default, it uses WebSocket and
listens to ``ws://localhost:9000``, but you can also point it to the file
system. If you saved the documents in a folder (using the folder export argument
of the Simple Producer), you can start it with the location of the manifest
file, like so: ``ebu-simple-consumer --manifest-path
myFolder/manifest_sequence_identifier.txt``.

User Input Producer
-------------------
This is a web page that adds a user interface and various configurations to the
Simple Producer. First, start a consumer or a handover node node
(``ebu-user-input-consumer``, ``ebu-user-input-forwarder`` or a handover node
from the command line. Then, in your browser, open
``ebu_tt_live/ui/user_input_producer/user_input_producer.html`` and click
'Connect'. Select the sending mode (manual, scheduled or asynchronous). You
should see the documents arriving in the command line window where
``ebu-user-input-consumer`` is listening. See detailed instructions here:
:doc:`user_input_producer`.

User Input Consumer
-------------------
This is very similar to the Simple Consumer. It also performs validation but it
does not have a manifest option. It provides a WebSocket connection point for
the User Input Producer. You'll need to start either this or the User Input
Forwarder before connecting the User Input Producer (but not both).

User Input Forwarder
--------------------
This script mimics a distribution node. It listens to documents coming from the
User Input Producer on ``ws://127.0.0.1:9001`` and forwards them to any consumer
listening on ``ws://127.0.0.1:9000``. Like the Simple Producer, it can also save
the documents it receives to the file system. First, run it with the
``--folder-export`` argument like this: ``ebu-user-input-forwarder
--folder-export myFolder``. Then launch the User Input Producer and connect. The
sequence will be saved to ``myFolder`` along with the manifest file. The User
Input Forwarder can also be used as an incoming connection point for WebSocket
connections from sources other than the User Input Producer.

Handover Manager
----------------
This node implements the 'Who claimed control most recently' algorithm defined
in the specification, with added functionality to allow messages to be broadcast
between members of the authors group, which may be added to a future
specification. This algorithm determines the output from multiple input
sequences. The Handover Manager is a specialised case of the switching node that
bases its decisions on handover-related attributes in the document and its
previous decisions. There is no separate command to run this script. Start it
with the ``ebu-run``, for example ``ebu-run
--admin.conf=ebu_tt_live/examples/config/user_input_producer_handover.json`` for
the default configuration. For detailed instruction on setting up the Handover
Manager with the UIP see :doc:`user_input_producer`.

EBU-TT-D Encoder
----------------
This script is an extension of simple consumer and is responsible for
resegmenting and converting the incoming EBU-TT Live documents into EBU-TT-D
documents that can be later used to be embedded in video streams such as DASH.
The script launches with the ``ebu-ebuttd-encoder`` command. There are switches
to control the mediatime conversion reference point and the segmentation
interval. Please run ``ebu-ebuttd-encoder --help`` to find our more.

The ebu-run script
------------------
This script is capable of picking up a structured configuration file and use
that to create the nodes and carriage mechanism objects that we would like to
use and they can be even wired in the same configuration file together so in the
end a complex system can be modeled using a single json file. Please refer to
:py:mod:`ebu_tt_live.scripts.ebu_run` as well as :py:mod:`ebu_tt_live.config` to
learn more about the way the configuration logic works. To get help on permitted
options for the specified node(s) run ``ebu-run`` with a ``--help``. See
:doc:`configurator`.
{'fraction_non_alphanumeric': 0.09168580615525954, 'fraction_numerical': 0.006982085438677079, 'mean_word_length': 4.216099664590321, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7380897', 'n_tokens_mistral': 2759, 'n_tokens_neox': 2652, 'n_words': 1485}