<commit_before>..
   SPDX-FileCopyrightText: 2021 <NAME>
   SPDX-License-Identifier: AGPL-3.0-or-later

=======
EventDB
=======

The EventDB is not a software itself.

The EventDB is a database (usually PostgreSQL) that gets filled with with data from intelmq using the SQL Output bot.

-----------------
EventDB Utilities
-----------------

Some scripts related to the eventdb are located in the `contrib/eventdb <https://github.com/certtools/intelmq/tree/develop/contrib/eventdb>`_ folder in the IntelMQ git repository.

Apply Malware Name Mapping
--------------------------

The `apply_mapping_eventdb.py` script applies the malware name mapping to the eventdb.
Source and destination columns can be given, also a local file. If no local file is present, the mapping can be downloaded on demand.<commit_msg>DOC: eventdb: describe the events table itself<commit_after>..
   SPDX-FileCopyrightText: 2021 <NAME>
   SPDX-License-Identifier: AGPL-3.0-or-later

=======
EventDB
=======

The EventDB is not a software itself.

The EventDB is a database (usually PostgreSQL) that gets filled with with data from intelmq using the SQL Output bot.

-----------------------
The events table itself
-----------------------

IntelMQ comes with the ``intelmq_psql_initdb`` command line tool. It creates an SQL file containing:

- A ``CREATE TABLE events`` statement with all valid IntelMQ fields as columns and correct types
- Several indexes as examples for a good read & search performance

All elements of this SQL file can be adapted and extended before running the SQL file against a database, especially the indexes.

Having an `events` table as outlined in the SQL file, IntelMQ's PostgreSQL Output bot can write all received events into this database table.

This events table is the core of the so-called EventDB and also required by all other sections of this document.

-----------------
EventDB Utilities
-----------------

Some scripts related to the eventdb are located in the `contrib/eventdb <https://github.com/certtools/intelmq/tree/develop/contrib/eventdb>`_ folder in the IntelMQ git repository.

Apply Malware Name Mapping
--------------------------

The `apply_mapping_eventdb.py` script applies the malware name mapping to the eventdb.
Source and destination columns can be given, also a local file. If no local file is present, the mapping can be downloaded on demand.
{'fraction_non_alphanumeric': 0.12851742965140697, 'fraction_numerical': 0.00503989920201596, 'mean_word_length': 3.171628721541156, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1962516', 'n_tokens_mistral': 624, 'n_tokens_neox': 588, 'n_words': 299}