<commit_before>How to document plugins
=======================

Background
----------

Plugin documentation happens in the source code of the plugin and is processed in two places.
First there is the godocs based documentation which is meant to be used by users with a programming background.
Second there is the read the docs based documentation which is meant to be used by users with an operational background.
Both documentations are generated from the comments in the source code.
The read the docs documentation is generated by a custom tool that converts the godoc formatted comments into RST.
As of this, the rule is to write everything godoc compatible but follow the rules of the RST generator.

Structure
---------

1) Heading
   A heading is defined by text that is surrounded by blank lines.
   If a heading is starting a documentation block, the preceding blank line is omitted.

2) Enumeration
   An enumeration starts with a "- " and ends with a blank line.

3) Keyed enumerations
   Keyed enumartions start with a "- " followed by the key, followed by a colon and end with a blank line.

Sections
--------

A plugin documentation is separated into 4 parts.

1) Explaining what the plugin does.
   The heading of this section starts with the name of the plugin followed by the type of the plugin, e.g. "Console consumer".
   The contents of this section should roughly describe what the plugin does plus any specialities that are important.

2) Explaining what metadata is consumed or generated
   The heading of this secion is "Metadata".
   The content of this section is an enumeration of all fields that are affected.
   If a field has a sepcific key it should be named first, followed by a colon, e.g. "- key: stores the key of the request".
   The RST generator will merge this section by traversing the type hierarchy of the plugin.
   Documentation from derived types will overwrite documentation of base types.

3) Explaining which parameters are supported
   The heading of this secion is "Parameters".
   The content of this section is an enumeration of all parameters that are defined in this plugin.
   Parameters defined by aggregated types are not listed except when changing the original definition.
   Parameters should start with the name, followed by a colon and end with the sentence "By default this parameter is set to <value>", e.g. "- servers: defines the list of servers to write to. By default this parameter is set to an empty list.".
   The RST generator will merge this section by traversing the type hierarchy of the plugin.
   Documentation from derived types will overwrite documentation of base types.

4) Giving examples
   The heading of this section is "Examples".
   This section should contain at least one configuration example.
   The example has to be meaningful and does not need to contain all parameters.
   The example has to be working, i.e. a user could copy-paste this to a config and it would work.
   The example is not complete, i.e. it only contains a configuration for this plugin not other plugins.
   If the plugin is a filter oder formatter (a nested plugin) the following stub should be used:

   ::

     ExampleConsumer:
       Type: consumer.Console
       Streams: console
       Modulators:

Example
-------

::

  // Console consumer:
  //
  // This consumer reads from stdin or a named pipe. A message is generated after
  // each newline character.
  //
  // Metadata:
  //
  // - pipe: name of the pipe the message was received on
  //
  // Parameters:
  //
  // - Pipe: Defines the pipe to read from. This can be "stdin" or the path
  // of a named pipe. A named pipe is creared if not existing.
  // The default is "stdin".
  //
  // - Permissions: Accepts an octal number string containing the unix file
  // permissions used when creating a named pipe.
  // By default this is set to "0664".
  //
  // - ExitOnEOF: Can be set to true to trigger an exit signal if the pipe is closed
  // i.e. when EOF is detected. This is set to true by default.
  //
  // Configuration example:
  //
  //  ConsoleIn:
  //    Type: consumer.Console
  //    Streams: console
  //    Pipe: /tmp/namedpipe
  //    Permissions: "0664"
  //    ExitOnEOF: false<commit_msg>add rules to make godoc more readable<commit_after>How to document plugins
=======================

Background
----------

Plugin documentation happens in the source code of the plugin and is processed in two places.
First there is the godocs based documentation which is meant to be used by users with a programming background.
Second there is the read the docs based documentation which is meant to be used by users with an operational background.
Both documentations are generated from the comments in the source code.
The read the docs documentation is generated by a custom tool that converts the godoc formatted comments into RST.
As of this, the rule is to write everything godoc compatible but follow the rules of the RST generator.

Structure
---------

1) Heading
   A heading is defined by text that is surrounded by blank lines.
   If a heading is starting a documentation block, the preceding blank line is omitted.
   Please note that a heading is only detected as heading by godoc if there is regular text following the blank line after the heading.

2) Enumeration
   An enumeration starts with a "- " and ends with a blank line.

3) Keyed enumerations
   Keyed enumartions start with a "- " followed by the key, followed by a colon and end with a blank line.

Sections
--------

A plugin documentation is separated into 4 parts.

1) Explaining what the plugin does.
   The heading of this section starts with the name of the plugin followed by the type of the plugin, e.g. "Console consumer".
   The contents of this section should roughly describe what the plugin does plus any specialities that are important.

2) Explaining what metadata is consumed or generated
   The heading of this secion is "Metadata".
   The content of this section is an enumeration of all fields that are affected.
   If a field has a sepcific key it should be named first, followed by a colon, e.g. "- key: stores the key of the request".
   The RST generator will merge this section by traversing the type hierarchy of the plugin.
   Documentation from derived types will overwrite documentation of base types.

3) Explaining which parameters are supported
   The heading of this secion is "Parameters".
   The content of this section is an enumeration of all parameters that are defined in this plugin.
   Parameters defined by aggregated types are not listed except when changing the original definition.
   Parameters should start with the name, followed by a colon and end with the sentence "By default this parameter is set to <value>", e.g. "- servers: defines the list of servers to write to. By default this parameter is set to an empty list.".
   The RST generator will merge this section by traversing the type hierarchy of the plugin.
   Documentation from derived types will overwrite documentation of base types.

4) Giving examples
   The heading of this section is "Examples".
   This section should contain at least one configuration example and a short description for each example provided.
   The example has to be meaningful and does not need to contain all parameters.
   The example has to be working, i.e. a user could copy-paste this to a config and it would work.
   The example is not complete, i.e. it only contains a configuration for this plugin not other plugins.
   If the plugin is a filter oder formatter (a nested plugin) the following stub should be used:

   ::

     ExampleConsumer:
       Type: consumer.Console
       Streams: console
       Modulators:

Example
-------

::

  // Console consumer
  //
  // This consumer reads from stdin or a named pipe. A message is generated after
  // each newline character.
  //
  // Metadata
  //
  // - pipe: name of the pipe the message was received on
  //
  // Parameters
  //
  // - Pipe: Defines the pipe to read from. This can be "stdin" or the path
  // of a named pipe. A named pipe is creared if not existing.
  // The default is "stdin".
  //
  // - Permissions: Accepts an octal number string containing the unix file
  // permissions used when creating a named pipe.
  // By default this is set to "0664".
  //
  // - ExitOnEOF: Can be set to true to trigger an exit signal if the pipe is closed
  // i.e. when EOF is detected. This is set to true by default.
  //
  // Examples
  //
  // This configuration reads data from standard-in.
  //
  //  ConsoleIn:
  //    Type: consumer.Console
  //    Streams: console
  //    Pipe: stdin
{'fraction_non_alphanumeric': 0.05900011546010853, 'fraction_numerical': 0.0032328830389100564, 'mean_word_length': 3.10521327014218, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '89733', 'n_tokens_mistral': 2151, 'n_tokens_neox': 2092, 'n_words': 1314}