<commit_before>Configuration options
=====================

The ROS build farm configuration has a similar structure than the rosdistro
specification (see `REP 143 <http://www.ros.org/reps/rep-0143.html>`_ for
details).

The configuration for the official ROS build farm configuration acts as an
example and can be found on
`GitHub (ros-infrastructure/ros_buildfarm_config) <https://github.com/ros-infrastructure/ros_buildfarm_config>`_.


Entry point yaml
----------------

This yaml file acts as the entry point to the ROS build farm configuration.
Most script require to pass the URL to this file as an argument.

The file format is specified by the following information:

* ``type: buildfarm`` identifies the yaml file as the entry point for the ROS
  build farm configuration.
* ``version: 1`` specifies the specification version of the file.

The following options are valid in version ``1``:

* ``distributions``: a dictionary mapping lowercase ROS distribution names to
  the following build files.
  The key "*default*" for a build file is hidden when being used to generate
  view / job prefixes.

  * ``release_builds``: a dictionary of release build files indexed by their
    name.
  * ``source_builds``: a dictionary of source build files indexed by their
    name.
  * ``doc_builds``: a dictionary of doc build files indexed by their name.

  All build file references can be either relative path or absolute URLs.

  * ``notification_emails``: a list of email addresses for notification about
    ROS distribution specific administrative jobs.

* ``jenkins_url``: the URL of the Jenkins master.

* ``notification_emails``: a list of email addresses for notification about
  ROS distribution agnostic administrative jobs.

* ``prerequisites``: a dictionary containing information about repositories
  which contain the necessary tools to run the ROS build farm.
  Currently the configuration only supports Debian repositories:

  * ``debian_repositories``: a list of URLs to Debian repositories.
  * ``debian_repository_keys``: a list of PGP keys each as a multi line string.
    The order of the keys must match the repository URLs.

* ``rosdistro_index_url``: the URL to the rosdistro index yaml file.
* ``status_page_repositories``: a dictionary mapping names to a list of
  repository URLs.
  For each entry in the dictionary a
  `repos-status-page <jobs/miscellaneous_jobs#status-pages.rst>`_ is being
  generated.


Generic options in build files
------------------------------

A set of options which can be used in any build file.

* ``repositories``: additional repositories to fetch packages from.

  * ``keys``: a list of PGP keys each as a multi line string.
    The order of the keys must match the repository URLs.
  * ``urls``: a list of URLs to repositories.

* ``tag_whitelist``: a list of tags to whitelist to identify which rosdistro
  distribution files this build file should operate on.
* ``tag_blacklist``: a list of tags to blacklist to identify which rosdistro
  distribution files this build file should not operate on.

* ``targets``: a nested dictionary of targets for which packages are built.
  The first level key contains the OS name.
  The second level key contains the OS code name.
  The third level key contains the CPU architecture.
  The third level value is always empty.

  The OS names and OS code names specified must be listed as a
  *release platform* in the corresponding rosdistro distribution file.

  A doc build file must only contain a single target.


Description of common options
-----------------------------

* **Jenkins job priorities**: a lower number means a higher priority.

  If no value is provided the generated job will not have specific priority.
  Then you should specify them using *JobGroups* through the Jenkins user
  interface.

  The job priorities of the administrative jobs are hard coded:

  * ``rosdistro-cache`` jobs: 10
  * ``check_slaves`` job: 20
  * ``dashboard`` job: 20
  * ``*-status-page`` jobs: 20
  * ``reconfigure-jobs`` jobs: 30
  * ``*-trigger-jobs`` jobs: 40

  It is highly recommended to only use lower priorities for the actual jobs
  (aka higher numbers).

* **Jenkins job timeouts**: the number of minutes after which a job should be
  aborted.

* **Jenkins job label**: the label expression to restrict the execution of a
  job to a specific computer or group of machines.

* **Notification emails**: the notification about repository / package specific
  job statuses can be send to:

  * a list of statically configure email addresses.
  * a list of maintainer email addresses extracted from the package manifests
    being processed.
  * a list of developer email addresses extracted from version control system.

* **Package / repository white- / blacklisting**: if no whitelist is specified
  all existing items are being used.
  The blacklist is applied after the whitelist which means if an item is in
  both lists it is being *excluded*.

* **Skip ignored packages / repositories**: by default jobs are still being
  generated for blacklisted (or not whitelisted) items but these jobs are
  disabled.
  To avoid generating these jobs to be generated set this flag to ``true``.

* **Credential ID**: the ID of the credential entry managed on the Jenkins
  master which is commonly used to upload artifacts to another host.


Specific options in release build files
---------------------------------------<commit_msg>Reference buildfarm_deployment for credential ID.<commit_after>Configuration options
=====================

The ROS build farm configuration has a similar structure than the rosdistro
specification (see `REP 143 <http://www.ros.org/reps/rep-0143.html>`_ for
details).

The configuration for the official ROS build farm configuration acts as an
example and can be found on
`GitHub (ros-infrastructure/ros_buildfarm_config) <https://github.com/ros-infrastructure/ros_buildfarm_config>`_.


Entry point yaml
----------------

This yaml file acts as the entry point to the ROS build farm configuration.
Most script require to pass the URL to this file as an argument.

The file format is specified by the following information:

* ``type: buildfarm`` identifies the yaml file as the entry point for the ROS
  build farm configuration.
* ``version: 1`` specifies the specification version of the file.

The following options are valid in version ``1``:

* ``distributions``: a dictionary mapping lowercase ROS distribution names to
  the following build files.
  The key "*default*" for a build file is hidden when being used to generate
  view / job prefixes.

  * ``release_builds``: a dictionary of release build files indexed by their
    name.
  * ``source_builds``: a dictionary of source build files indexed by their
    name.
  * ``doc_builds``: a dictionary of doc build files indexed by their name.

  All build file references can be either relative path or absolute URLs.

  * ``notification_emails``: a list of email addresses for notification about
    ROS distribution specific administrative jobs.

* ``jenkins_url``: the URL of the Jenkins master.

* ``notification_emails``: a list of email addresses for notification about
  ROS distribution agnostic administrative jobs.

* ``prerequisites``: a dictionary containing information about repositories
  which contain the necessary tools to run the ROS build farm.
  Currently the configuration only supports Debian repositories:

  * ``debian_repositories``: a list of URLs to Debian repositories.
  * ``debian_repository_keys``: a list of PGP keys each as a multi line string.
    The order of the keys must match the repository URLs.

* ``rosdistro_index_url``: the URL to the rosdistro index yaml file.
* ``status_page_repositories``: a dictionary mapping names to a list of
  repository URLs.
  For each entry in the dictionary a
  `repos-status-page <jobs/miscellaneous_jobs#status-pages.rst>`_ is being
  generated.


Generic options in build files
------------------------------

A set of options which can be used in any build file.

* ``repositories``: additional repositories to fetch packages from.

  * ``keys``: a list of PGP keys each as a multi line string.
    The order of the keys must match the repository URLs.
  * ``urls``: a list of URLs to repositories.

* ``tag_whitelist``: a list of tags to whitelist to identify which rosdistro
  distribution files this build file should operate on.
* ``tag_blacklist``: a list of tags to blacklist to identify which rosdistro
  distribution files this build file should not operate on.

* ``targets``: a nested dictionary of targets for which packages are built.
  The first level key contains the OS name.
  The second level key contains the OS code name.
  The third level key contains the CPU architecture.
  The third level value is always empty.

  The OS names and OS code names specified must be listed as a
  *release platform* in the corresponding rosdistro distribution file.

  A doc build file must only contain a single target.


Description of common options
-----------------------------

* **Jenkins job priorities**: a lower number means a higher priority.

  If no value is provided the generated job will not have specific priority.
  Then you should specify them using *JobGroups* through the Jenkins user
  interface.

  The job priorities of the administrative jobs are hard coded:

  * ``rosdistro-cache`` jobs: 10
  * ``check_slaves`` job: 20
  * ``dashboard`` job: 20
  * ``*-status-page`` jobs: 20
  * ``reconfigure-jobs`` jobs: 30
  * ``*-trigger-jobs`` jobs: 40

  It is highly recommended to only use lower priorities for the actual jobs
  (aka higher numbers).

* **Jenkins job timeouts**: the number of minutes after which a job should be
  aborted.

* **Jenkins job label**: the label expression to restrict the execution of a
  job to a specific computer or group of machines.

* **Notification emails**: the notification about repository / package specific
  job statuses can be send to:

  * a list of statically configure email addresses.
  * a list of maintainer email addresses extracted from the package manifests
    being processed.
  * a list of developer email addresses extracted from version control system.

* **Package / repository white- / blacklisting**: if no whitelist is specified
  all existing items are being used.
  The blacklist is applied after the whitelist which means if an item is in
  both lists it is being *excluded*.

* **Skip ignored packages / repositories**: by default jobs are still being
  generated for blacklisted (or not whitelisted) items but these jobs are
  disabled.
  To avoid generating these jobs to be generated set this flag to ``true``.

* **Credential ID**: the ID of the credential entry managed on the Jenkins
  master which is commonly used to upload artifacts to another host.
  This credential id is set in the buildfarm_deployment.


Specific options in release build files
---------------------------------------
{'fraction_non_alphanumeric': 0.08590971272229822, 'fraction_numerical': 0.0038303693570451436, 'mean_word_length': 3.712505371723249, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3076183', 'n_tokens_mistral': 2878, 'n_tokens_neox': 2677, 'n_words': 1490}