<commit_before>..
   SPDX-FileCopyrightText: 2015 <NAME> <<EMAIL>>
   SPDX-License-Identifier: AGPL-3.0-or-later

############################
Configuration and Management
############################

.. contents::

For installation instructions, see :doc:`installation`.
For upgrade instructions, see :doc:`upgrade`.

******************
Where to get help?
******************

In case you are lost or something is not discussed in this guide, you might want to subscribe to the |intelmq-users-list-link| and ask your questions there.

With that clarified, let's dig into the details...


******************
Configure services
******************
You need to enable and start Redis if not already done. Using systemd it can be done with:

.. code-block:: bash

   systemctl enable redis.service
   systemctl start redis.service

.. _configuration:

*************
Configuration
*************

/opt and LSB paths
==================

If you installed the packages, standard Linux paths (LSB paths) are used: ``/var/log/intelmq/``, ``/etc/intelmq/``, ``/var/lib/intelmq/``, ``/var/run/intelmq/``.
Otherwise, the configuration directory is ``/opt/intelmq/etc/``. Using the environment variable ``INTELMQ_ROOT_DIR`` allows setting any arbitrary root directory.

You can switch this by setting the environment variables ``INTELMQ_PATHS_NO_OPT`` and ``INTELMQ_PATHS_OPT``, respectively.
* When installing the Python packages, you can set ``INTELMQ_PATHS_NO_OPT`` to something non-empty to use LSB-paths.
* When installing the deb/rpm packages, you can set ``INTELMQ_PATHS_OPT`` to something non-empty to use ``/opt/intelmq/`` paths, or a path set with ``INTELMQ_ROOT_DIR``.

The environment variable ``ROOT_DIR`` is meant to set an alternative root directory instead of `/`. This is primarily meant for package build environments an analogous to setuptools' ``--root`` parameter. Thus it is only used in LSB-mode.

Overview
========

The main configuration file is formatted in the YAML format since IntelMQ 3.0 (before it was JSON, which had some downsides).
For new installations a default setup with some examples is provided by the `intelmqsetup` tool. If this is not the case, make sure the program was run (see installation instructions).


* ``runtime.yaml``: Configuration for the individual bots. See :doc:`bots` for more details.

To configure a new bot, you need to define and configure it in ``runtime.yaml``. You can base your configuration on the output of ``intelmqctl list bots``.
Use the IntelMQ Manager mentioned above to generate the configuration files if unsure.

In the shipped examples 4 collectors and parsers, 6 common experts and one output are configured. The default collector and the parser handle data from malware domain list, the file output bot writes all data to ``/opt/intelmq/var/lib/bots/file-output/events.txt``/``/var/lib/intelmq/bots/file-output/events.txt``.

*********************************
Systemwide Configuration (global)
*********************************

All bots inherit the global configuration parameters in the ``runtime.conf`` and they can overwrite them using the same parameters in their individual configuration in the ``runtime.conf`` file.

.. _configuration-logging:

Logging
=======

The logging can be configured with the following parameters:

* ``logging_handler``: Can be one of ``"file"`` or ``"syslog"``.
* ``logging_level``: Defines the system-wide log level that will be use by all bots and the intelmqctl tool. Possible values are: ``"CRITICAL"``, ``"ERROR"``, ``"WARNING"``, ``"INFO"`` and ``"DEBUG"``.
* ``logging_path``: If ``logging_handler`` is ``file``. Defines the system-wide log-folder that will be use by all bots and the intelmqctl tool. Default value: ``/opt/intelmq/var/log/`` or ``/var/log/intelmq/`` respectively.
* ``logging_syslog``: If ``logging_handler`` is ``syslog``. Either a list with hostname and UDP port of syslog service, e.g. ``["localhost", 514]`` or a device name/path, e.g. the default ``"/var/log"``.

We recommend ``logging_level`` ``WARNING`` for production environments and ``INFO`` if you want more details. In any case, watch your free disk space!

Log rotation
------------
<commit_msg>doc: update configuration overview section<commit_after>..
   SPDX-FileCopyrightText: 2015 <NAME> <<EMAIL>>, 2015-2021 <NAME>, 2020-2021 <NAME>
   SPDX-License-Identifier: AGPL-3.0-or-later

############################
Configuration and Management
############################

.. contents::

For installation instructions, see :doc:`installation`.
For upgrade instructions, see :doc:`upgrade`.

******************
Where to get help?
******************

In case you are lost or something is not discussed in this guide, you might want to subscribe to the |intelmq-users-list-link| and ask your questions there.

With that clarified, let's dig into the details...


******************
Configure services
******************
You need to enable and start Redis if not already done. Using systemd it can be done with:

.. code-block:: bash

   systemctl enable redis.service
   systemctl start redis.service

.. _configuration:

*************
Configuration
*************

/opt and LSB paths
==================

If you installed the packages, standard Linux paths (LSB paths) are used: ``/var/log/intelmq/``, ``/etc/intelmq/``, ``/var/lib/intelmq/``, ``/var/run/intelmq/``.
Otherwise, the configuration directory is ``/opt/intelmq/etc/``. Using the environment variable ``INTELMQ_ROOT_DIR`` allows setting any arbitrary root directory.

You can switch this by setting the environment variables ``INTELMQ_PATHS_NO_OPT`` and ``INTELMQ_PATHS_OPT``, respectively.
* When installing the Python packages, you can set ``INTELMQ_PATHS_NO_OPT`` to something non-empty to use LSB-paths.
* When installing the deb/rpm packages, you can set ``INTELMQ_PATHS_OPT`` to something non-empty to use ``/opt/intelmq/`` paths, or a path set with ``INTELMQ_ROOT_DIR``.

The environment variable ``ROOT_DIR`` is meant to set an alternative root directory instead of `/`. This is primarily meant for package build environments an analogous to setuptools' ``--root`` parameter. Thus it is only used in LSB-mode.

Overview
========

The main configuration file is formatted in the YAML format since IntelMQ 3.0 (before it was JSON, which had some downsides).
Although, comments in YAML are currently not preserved by IntelMQ (known bug `#2003 <https://github.com/certtools/intelmq/issues/2003>`_).
For new installations a default setup with some examples is provided by the `intelmqsetup` tool. If this is not the case, make sure the program was run (see :doc:`installation` instructions).


* ``runtime.yaml``: Configuration for the individual bots. See :doc:`bots` for more details.
* ``harmonization.conf``: Configuration of the internal data format, see :doc:`/dev/data-format` and :doc:`/dev/harmonization-fields`.

To configure a new bot, you need to define and configure it in ``runtime.yaml``.
You can base your configuration on the output of ``intelmqctl list bots`` and the :doc:`feeds` documentation page.
Use the IntelMQ Manager mentioned above to generate the configuration files if unsure.

In the shipped examples 4 collectors and parsers, 6 common experts and one output are configured. The default collector and the parser handle data from malware domain list, the file output bot writes all data to ``/opt/intelmq/var/lib/bots/file-output/events.txt``/``/var/lib/intelmq/bots/file-output/events.txt``.

*********************************
Systemwide Configuration (global)
*********************************

All bots inherit the global configuration parameters in the ``runtime.conf`` and they can overwrite them using the same parameters in their individual configuration in the ``runtime.conf`` file.

.. _configuration-logging:

Logging
=======

The logging can be configured with the following parameters:

* ``logging_handler``: Can be one of ``"file"`` or ``"syslog"``.
* ``logging_level``: Defines the system-wide log level that will be use by all bots and the intelmqctl tool. Possible values are: ``"CRITICAL"``, ``"ERROR"``, ``"WARNING"``, ``"INFO"`` and ``"DEBUG"``.
* ``logging_path``: If ``logging_handler`` is ``file``. Defines the system-wide log-folder that will be use by all bots and the intelmqctl tool. Default value: ``/opt/intelmq/var/log/`` or ``/var/log/intelmq/`` respectively.
* ``logging_syslog``: If ``logging_handler`` is ``syslog``. Either a list with hostname and UDP port of syslog service, e.g. ``["localhost", 514]`` or a device name/path, e.g. the default ``"/var/log"``.

We recommend ``logging_level`` ``WARNING`` for production environments and ``INFO`` if you want more details. In any case, watch your free disk space!

Log rotation
------------

{'fraction_non_alphanumeric': 0.16160347190497945, 'fraction_numerical': 0.005710370031978072, 'mean_word_length': 4.88508064516129, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3614972', 'n_tokens_mistral': 2482, 'n_tokens_neox': 2344, 'n_words': 1042}