<commit_before>=======================
Ganeti monitoring agent
=======================

.. contents:: :depth: 4

This is a design document detailing the implementation of a Ganeti
monitoring agent report system, that can be queried by a monitoring
system to calculate health information for a Ganeti cluster.

Current state and shortcomings
==============================

There is currently no monitoring support in Ganeti. While we don't want
to build something like Nagios or Pacemaker as part of Ganeti, it would
be useful if such tools could easily extract information from a Ganeti
machine in order to take actions (example actions include logging an
outage for future reporting or alerting a person or system about it).

Proposed changes
================

Each Ganeti node should export a status page that can be queried by a
monitoring system. Such status page will be exported on a network port
and will be encoded in JSON (simple text) over HTTP.

The choice of json is obvious as we already depend on it in Ganeti and
thus we don't need to add extra libraries to use it, as opposed to what
would happen for XML or some other markup format.

Location of agent report
------------------------

The report will be available from all nodes, and be concerned for all
node-local resources. This allows more real-time information to be
available, at the cost of querying all nodes.

Information reported
--------------------

The monitoring agent system will report on the following basic information:

- Instance status
- Instance disk status
- Status of storage for instances
- Ganeti daemons status, CPU usage, memory footprint
- Hypervisor resources report (memory, CPU, network interfaces)
- Node OS resources report (memory, CPU, network interfaces)
- Information from a plugin system

Instance status
+++++++++++++++

At the moment each node knows which instances are running on it, which
instances it is primary for, but not the cause why an instance might not
be running. On the other hand we don't want to distribute full instance
"admin" status information to all nodes, because of the performance
impact this would have.

As such we propose that:

- Any operation that can affect instance status will have an optional
  "reason" attached to it (at opcode level). This can be used for
  example to distinguish an admin request, from a scheduled maintenance
  or an automated tool's work. If this reason is not passed, Ganeti will
  just use the information it has about the source of the request: for
  example a cli shutdown operation will have "cli:shutdown" as a reason,
  a cli failover operation will have "cli:failover". Operations coming
  from the remote API will use "rapi" instead of "cli". Of course
  setting a real site-specific reason is still preferred.
- RPCs that affect the instance status will be changed so that the
  "reason" and the version of the config object they ran on is passed to
  them. They will then export the new expected instance status, together
  with the associated reason and object version to the status report
  system, which then will export those themselves.

Monitoring and auditing systems can then use the reason to understand
the cause of an instance status, and they can use the object version to
understand the freshness of their data even in the absence of an atomic
cross-node reporting: for example if they see an instance "up" on a node
after seeing it running on a previous one, they can compare these values
to understand which data is freshest, and repoll the "older" node. Of
course if they keep seeing this status this represents an error (either
an instance continuously "flapping" between nodes, or an instance is
constantly up on more than one), which should be reported and acted
upon.

The instance status will be on each node, for the instances it is
primary for and will contain at least:

- The instance name
- The instance UUID (stable on name change)
- The instance running status (up or down)
- The timestamp of last known change
- The timestamp of when the status was last checked (see caching, below)
- The last known reason for change, if any

More information about all the fields and their type will be available
in the "Format of the report" section.

Note that as soon as a node knows it's not the primary anymore for an
instance it will stop reporting status for it: this means the instance<commit_msg>Add instance uptime to monitoring info<commit_after>=======================
Ganeti monitoring agent
=======================

.. contents:: :depth: 4

This is a design document detailing the implementation of a Ganeti
monitoring agent report system, that can be queried by a monitoring
system to calculate health information for a Ganeti cluster.

Current state and shortcomings
==============================

There is currently no monitoring support in Ganeti. While we don't want
to build something like Nagios or Pacemaker as part of Ganeti, it would
be useful if such tools could easily extract information from a Ganeti
machine in order to take actions (example actions include logging an
outage for future reporting or alerting a person or system about it).

Proposed changes
================

Each Ganeti node should export a status page that can be queried by a
monitoring system. Such status page will be exported on a network port
and will be encoded in JSON (simple text) over HTTP.

The choice of json is obvious as we already depend on it in Ganeti and
thus we don't need to add extra libraries to use it, as opposed to what
would happen for XML or some other markup format.

Location of agent report
------------------------

The report will be available from all nodes, and be concerned for all
node-local resources. This allows more real-time information to be
available, at the cost of querying all nodes.

Information reported
--------------------

The monitoring agent system will report on the following basic information:

- Instance status
- Instance disk status
- Status of storage for instances
- Ganeti daemons status, CPU usage, memory footprint
- Hypervisor resources report (memory, CPU, network interfaces)
- Node OS resources report (memory, CPU, network interfaces)
- Information from a plugin system

Instance status
+++++++++++++++

At the moment each node knows which instances are running on it, which
instances it is primary for, but not the cause why an instance might not
be running. On the other hand we don't want to distribute full instance
"admin" status information to all nodes, because of the performance
impact this would have.

As such we propose that:

- Any operation that can affect instance status will have an optional
  "reason" attached to it (at opcode level). This can be used for
  example to distinguish an admin request, from a scheduled maintenance
  or an automated tool's work. If this reason is not passed, Ganeti will
  just use the information it has about the source of the request: for
  example a cli shutdown operation will have "cli:shutdown" as a reason,
  a cli failover operation will have "cli:failover". Operations coming
  from the remote API will use "rapi" instead of "cli". Of course
  setting a real site-specific reason is still preferred.
- RPCs that affect the instance status will be changed so that the
  "reason" and the version of the config object they ran on is passed to
  them. They will then export the new expected instance status, together
  with the associated reason and object version to the status report
  system, which then will export those themselves.

Monitoring and auditing systems can then use the reason to understand
the cause of an instance status, and they can use the object version to
understand the freshness of their data even in the absence of an atomic
cross-node reporting: for example if they see an instance "up" on a node
after seeing it running on a previous one, they can compare these values
to understand which data is freshest, and repoll the "older" node. Of
course if they keep seeing this status this represents an error (either
an instance continuously "flapping" between nodes, or an instance is
constantly up on more than one), which should be reported and acted
upon.

The instance status will be on each node, for the instances it is
primary for and will contain at least:

- The instance name
- The instance UUID (stable on name change)
- The instance running status (up or down)
- The uptime, as detected by the hypervisor
- The timestamp of last known change
- The timestamp of when the status was last checked (see caching, below)
- The last known reason for change, if any

More information about all the fields and their type will be available
in the "Format of the report" section.

Note that as soon as a node knows it's not the primary anymore for an
instance it will stop reporting status for it: this means the instance
{'fraction_non_alphanumeric': 0.0632268262391329, 'fraction_numerical': 0.00022581009371118888, 'mean_word_length': 4.171044950379451, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3079702', 'n_tokens_mistral': 2089, 'n_tokens_neox': 2008, 'n_words': 1407}