<commit_before>.. _control:

Control bridges
===============

The **livebridge** command expects a **-control=** parameter, specifying the path to a control file with the \
configuration for the content bridges.

**--control** can be either a **local control file** or a **remote control file** on **s3**:

* with local control file

.. code-block:: bash

    livebridge --control=/path/to/control.yaml

* with remote control file on S3

.. code-block:: bash

    livebridge --control=s3://bucketname/control.yaml

.. note::
  The control file can alternatively be specified via the **LB_CONTROLFILE** environment variable.


The bridges, over which the content is syndicated, are setup in this control file in *YAML* format. Use \
`control.yaml.example`_ as basis for your own control file.

.. code-block:: yaml

    auth:
        dev:
            user: "dev"
            password: "<PASSWORD>"
            api_key: "F00Baz"
        live:
            user: "prod"
            password: "<PASSWORD>"
            api_key: "Foobar"
    bridges:
        - source_id: "abcdefg"
          endpoint:  "https://example.com/api/"
          label: "Example 1"
          targets:
            - type: "scribble"
              event_id: "123456"
              auth: "dev"
            - type: "another"
              event_id: "654321"
              auth: "live"
        - source_id: 54321
          endpoint:  "https://foo.org/api/"
          label: "Example 2"
          targets:
            - type: "scribble"
              event_id: "123456"
              auth: "dev"

Authorization
-------------
    
Credentials for service authorization are defined under **auth:**, in which the keys \
like **dev** or **live** are later used for reference under **bridges:** and **targets:**. 
<commit_msg>Docs: Add information about control-data from storage.<commit_after>.. _control:

Control bridges
===============

The bridges will be specified by extra control data, which can be stored in an extra .yaml control file \
or as JSON in a database table.

Control data in .yaml file
--------------------------

The **livebridge** command expects a **-control=** parameter or alternatively the **LB_CONTROLFILE** environment var. \
It specifies the path to a control file with the configuration for the content bridges.

**--control / LB_CONTROLFILE** can be either a **local control file** or a **remote control file** on **s3**:

* with local control file

.. code-block:: bash

    livebridge --control=/path/to/control.yaml

* with remote control file on S3

.. code-block:: bash

    livebridge --control=s3://bucketname/control.yaml

* with remote file on S3, specified via environment variable

.. code-block:: bash

    LB_CONTROLFILE=s3://bucketname/control.yaml livebridge

The bridges, over which the content is syndicated, are setup in this control file in *YAML* format. Use \
`control.yaml.example`_ as basis for your own control file.

.. code-block:: yaml

    auth:
        dev:
            user: "dev"
            password: "<PASSWORD>"
            api_key: "F00Baz"
        live:
            user: "prod"
            password: "<PASSWORD>"
            api_key: "Foobar"
    bridges:
        - source_id: "abcdefg"
          endpoint:  "https://example.com/api/"
          label: "Example 1"
          targets:
            - type: "scribble"
              event_id: "123456"
              auth: "dev"
            - type: "another"
              event_id: "654321"
              auth: "live"
        - source_id: 54321
          endpoint:  "https://foo.org/api/"
          label: "Example 2"
          targets:
            - type: "scribble"
              event_id: "123456"
              auth: "dev"


Control data stored in database
-------------------------------

The configuration for bridges can also be saved in the database storage used. This can be either a SQL-based database supported by SQLALchemy_ or `AWS DynamoDB`_.

The name of the table has to be specified by an environment var, either by **LB_DB_CONTROL_TABLE** (SQL) or **LB_DYNAMO_CONTROL_TABLE** (DynamoDB). The table will be created, if it does not exist.

The data is stored as **JSON** and looks correspondent to the example above the following way:

.. code-block:: json

    {
        "auth": {
            "dev": {
                "user": "dev",
                "password": "<PASSWORD>",
                "api_key": "F00Baz"
            },
            "live": {
                "user": "prod",
                "password": "<PASSWORD>",
                "api_key": "Foobar"
            }
        },
        "bridges": [
            {
                "source_id": "abcdefg",
                "endpoint":  "https://example.com/api/",
                "label": "Example 1",
                "targets": [
                    {
                        "type": "scribble",
                        "event_id": "123456",
                        "auth": "dev"
                    },{
                        "type": "another",
                        "event_id": "654321",
                        "auth": "live"
                   }
                ]
            },{
                "source_id": 54321,
                "endpoint": "https://foo.org/api/",
                "label": "Example 2",
                "targets": [
                    {
                        "type": "scribble",
                        "event_id": "123456",
                        "auth": "dev"
                    }
                ]
            }
        ]
    }


Authorization
-------------

Credentials for service authorization are defined under **auth:**, in which the keys \
like **dev** or **live** are later used for reference under **bridges:** and **targets:**. 

{'fraction_non_alphanumeric': 0.1312565628281414, 'fraction_numerical': 0.015575778788939447, 'mean_word_length': 1.6470588235294117, 'pattern_counts': {'":': 27, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5684115', 'n_tokens_mistral': 1648, 'n_tokens_neox': 1500, 'n_words': 524}