<commit_before>.. _concepts:

------------------------
Concepts
------------------------

Buildbot provides an excellent base, but managing large number of bots is tedious. CthulhuBot (CB) provides further abstraction on top of buildbot and citools to ease creation and management of large number of projects, as well as better monitoring and reporting.

CB assumes you're controlling all parts of infrastrucutre and utilizes remote access using ssh and sudo privileges. It's not suitable for "buildslave-donated" environments.

CB is primarily written as managing continuous integration for developing django-based web application in debian environment.

For now, CB is assuming it runs on same machine as all buildmasters will be generated. Remote masters could be added in the future (aka fork me on github!).

------------------------
Vocabulary
------------------------

    CthulhuBot
        In this documentation, it's used mostly as "running project instance using django-massive-buildbot reusable application" instead of "python library".

    Project
        Represents either one application (in django sense, i.e. library) or project (meaning "deployed page"). Project can be depend on other and have responsible people associated. It has at least one buildmaster (this is tool for handling CI, not Trac).

    Build Computer
        Computer with POSIX operating system (nothing else supported for now; Debian have special flavor), remotely controllable by CthulhuBot. We're assuming it's dedicated for buildbots job and CthulhuBot is sudo-priveledged for some operations, called remotely by ssh with no-password key authentication.

    Buildmaster
        As in BuildBot; coordinating server program assigned set of buildslaves. It's using configuration generated by massive-buildbot and it's coordinated by it, have two dedicated port (coordination port for slaves and for waterfall page view).

    Command
        Step in buildbot dictionary. One (usually shell command) to be executed, as part of the job.

    Job
        In BuildBots speech: named factory. For normal people: sequence of commands with assigned name, so they can be used across projects.

    Job Assigment
        Job, configured for given Project to work on given Build Computer. In BuildBot's speech, this roughly matches builder.

    Project Client
        Client process running on Build Computer, connecting to correct Buildmaster port and listening to it's commands. Created automagically on every Build Computer that is working for a Project. BuildSlave if BuildBot speech.

------------------------
Software stages
------------------------

CB heavily uses Job, áka "named factory", i.e. abstration of top of buildbot's factory. Each named factory is just sequence of configurable commands, executed in order and bound by some rules (locks etc.). Important jobs are:

* package build (and upload and master ping)
* documentation build (and upload)
* test run
* webtests run (and configuration of live port and selenium proxy)


----------------------------------------------------
Dependency handling (aka applications and projects)
----------------------------------------------------

This is implemented in custom and hacky way and will be reimplemented by writing RPC plugin for buildbot.

-----------------------------------------------
Jobs and Commands creation and configuration
-----------------------------------------------

Commands are basically factories producing configurable buildbot steps. Difference is that steps command can populate configuration parameters. Parameter value could be specified either in Job in which Command is present, or in Project assigment (Parameters are then shown in CthulhuBot UI).

Both commands and jobs are pluggable using setuptools entry points, under groups "cthulhubot.commands" and "cthulhubot.jobs". To avoid :cmd:`setup.py develop` requirement, all classes from "cthulhubot.commands" and "chtulhubot.jobs" with defined "slug" attribute are also resolved.

Command can be configured when assigned to Job, althrough Job can decide not to specify the options, but configure them when Job itself will be configured. Job is configured when put to work, which means when associated to Project and BuildComputer. Using ERD, this could be modelled as follows:<commit_msg>Fixed heading levels<commit_after>.. _concepts:

------------------------
Concepts
------------------------

Buildbot provides an excellent base, but managing large number of bots is tedious. CthulhuBot (CB) provides further abstraction on top of buildbot and citools to ease creation and management of large number of projects, as well as better monitoring and reporting.

CB assumes you're controlling all parts of infrastrucutre and utilizes remote access using ssh and sudo privileges. It's not suitable for "buildslave-donated" environments.

CB is primarily written as managing continuous integration for developing django-based web application in debian environment.

For now, CB is assuming it runs on same machine as all buildmasters will be generated. Remote masters could be added in the future (aka fork me on github!).

=======================
Vocabulary
=======================

    CthulhuBot
        In this documentation, it's used mostly as "running project instance using django-massive-buildbot reusable application" instead of "python library".

    Project
        Represents either one application (in django sense, i.e. library) or project (meaning "deployed page"). Project can be depend on other and have responsible people associated. It has at least one buildmaster (this is tool for handling CI, not Trac).

    Build Computer
        Computer with POSIX operating system (nothing else supported for now; Debian have special flavor), remotely controllable by CthulhuBot. We're assuming it's dedicated for buildbots job and CthulhuBot is sudo-priveledged for some operations, called remotely by ssh with no-password key authentication.

    Buildmaster
        As in BuildBot; coordinating server program assigned set of buildslaves. It's using configuration generated by massive-buildbot and it's coordinated by it, have two dedicated port (coordination port for slaves and for waterfall page view).

    Command
        Step in buildbot dictionary. One (usually shell command) to be executed, as part of the job.

    Job
        In BuildBots speech: named factory. For normal people: sequence of commands with assigned name, so they can be used across projects.

    Job Assigment
        Job, configured for given Project to work on given Build Computer. In BuildBot's speech, this roughly matches builder.

    Project Client
        Client process running on Build Computer, connecting to correct Buildmaster port and listening to it's commands. Created automagically on every Build Computer that is working for a Project. BuildSlave if BuildBot speech.

=======================
Software stages
=======================

CB heavily uses Job, áka "named factory", i.e. abstration of top of buildbot's factory. Each named factory is just sequence of configurable commands, executed in order and bound by some rules (locks etc.). Important jobs are:

* package build (and upload and master ping)
* documentation build (and upload)
* test run
* webtests run (and configuration of live port and selenium proxy)


=====================================================
Dependency handling (aka applications and projects)
=====================================================

This is implemented in custom and hacky way and will be reimplemented by writing RPC plugin for buildbot.

=====================================================
Jobs and Commands creation and configuration
=====================================================

Commands are basically factories producing configurable buildbot steps. Difference is that steps command can populate configuration parameters. Parameter value could be specified either in Job in which Command is present, or in Project assigment (Parameters are then shown in CthulhuBot UI).

Both commands and jobs are pluggable using setuptools entry points, under groups "cthulhubot.commands" and "cthulhubot.jobs". To avoid :cmd:`setup.py develop` requirement, all classes from "cthulhubot.commands" and "chtulhubot.jobs" with defined "slug" attribute are also resolved.

Command can be configured when assigned to Job, althrough Job can decide not to specify the options, but configure them when Job itself will be configured. Job is configured when put to work, which means when associated to Project and BuildComputer. Using ERD, this could be modelled as follows:
{'fraction_non_alphanumeric': 0.11582476014333604, 'fraction_numerical': 0.0, 'mean_word_length': 3.6094832179009058, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4392131', 'n_tokens_mistral': 1961, 'n_tokens_neox': 1830, 'n_words': 1113}