<commit_before>=================
Ganeti 2.1 design
=================

This document describes the major changes in Ganeti 2.1 compared to
the 2.0 version.

The 2.1 version will be a relatively small release. Its main aim is to avoid
changing too much of the core code, while addressing issues and adding new
features and improvements over 2.0, in a timely fashion.

.. contents:: :depth: 3

Objective
=========

Ganeti 2.1 will add features to help further automatization of cluster
operations, further improbe scalability to even bigger clusters, and make it
easier to debug the Ganeti core.

Background
==========

Overview
========

Detailed design
===============

As for 2.0 we divide the 2.1 design into three areas:

- core changes, which affect the master daemon/job queue/locking or all/most
  logical units
- logical unit/feature changes
- external interface changes (eg. command line, os api, hooks, ...)

Core changes
------------

Storage units modelling
~~~~~~~~~~~~~~~~~~~~~~~

Currently, Ganeti has a good model of the block devices for instances
(e.g. LVM logical volumes, files, DRBD devices, etc.) but none of the
storage pools that are providing the space for these front-end
devices. For example, there are hardcoded inter-node RPC calls for
volume group listing, file storage creation/deletion, etc.

The storage units framework will implement a generic handling for all
kinds of storage backends:

- LVM physical volumes
- LVM volume groups
- File-based storage directories
- any other future storage method

There will be a generic list of methods that each storage unit type
will provide, like:

- list of storage units of this type
- check status of the storage unit

Additionally, there will be specific methods for each method, for example:

- enable/disable allocations on a specific PV
- file storage directory creation/deletion
- VG consistency fixing

This will allow a much better modeling and unification of the various
RPC calls related to backend storage pool in the future. Ganeti 2.1 is
intended to add the basics of the framework, and not necessarilly move
all the curent VG/FileBased operations to it.

Note that while we model both LVM PVs and LVM VGs, the framework will
**not** model any relationship between the different types. In other
words, we don't model neither inheritances nor stacking, since this is
too complex for our needs. While a ``vgreduce`` operation on a LVM VG
could actually remove a PV from it, this will not be handled at the
framework level, but at individual operation level. The goal is that
this is a lightweight framework, for abstracting the different storage
operation, and not for modelling the storage hierarchy.

Feature changes
---------------

Ganeti Confd
~~~~~~~~~~~~

Current State and shortcomings
++++++++++++++++++++++++++++++
In Ganeti 2.0 all nodes are equal, but some are more equal than others. In
particular they are divided between "master", "master candidates" and "normal".
(Moreover they can be offline or drained, but this is not important for the
current discussion). In general the whole configuration is only replicated to
master candidates, and some partial information is spread to all nodes via
ssconf.

This change was done so that the most frequent Ganeti operations didn't need to
contact all nodes, and so clusters could become bigger. If we want more
information to be available on all nodes, we need to add more ssconf values,
which is counter-balancing the change, or to talk with the master node, which
is not designed to happen now, and requires its availability.<commit_msg>Add locking improvements to Ganeti 2.1 design document<commit_after>=================
Ganeti 2.1 design
=================

This document describes the major changes in Ganeti 2.1 compared to
the 2.0 version.

The 2.1 version will be a relatively small release. Its main aim is to avoid
changing too much of the core code, while addressing issues and adding new
features and improvements over 2.0, in a timely fashion.

.. contents:: :depth: 4

Objective
=========

Ganeti 2.1 will add features to help further automatization of cluster
operations, further improbe scalability to even bigger clusters, and make it
easier to debug the Ganeti core.

Background
==========

Overview
========

Detailed design
===============

As for 2.0 we divide the 2.1 design into three areas:

- core changes, which affect the master daemon/job queue/locking or all/most
  logical units
- logical unit/feature changes
- external interface changes (eg. command line, os api, hooks, ...)

Core changes
------------

Storage units modelling
~~~~~~~~~~~~~~~~~~~~~~~

Currently, Ganeti has a good model of the block devices for instances
(e.g. LVM logical volumes, files, DRBD devices, etc.) but none of the
storage pools that are providing the space for these front-end
devices. For example, there are hardcoded inter-node RPC calls for
volume group listing, file storage creation/deletion, etc.

The storage units framework will implement a generic handling for all
kinds of storage backends:

- LVM physical volumes
- LVM volume groups
- File-based storage directories
- any other future storage method

There will be a generic list of methods that each storage unit type
will provide, like:

- list of storage units of this type
- check status of the storage unit

Additionally, there will be specific methods for each method, for example:

- enable/disable allocations on a specific PV
- file storage directory creation/deletion
- VG consistency fixing

This will allow a much better modeling and unification of the various
RPC calls related to backend storage pool in the future. Ganeti 2.1 is
intended to add the basics of the framework, and not necessarilly move
all the curent VG/FileBased operations to it.

Note that while we model both LVM PVs and LVM VGs, the framework will
**not** model any relationship between the different types. In other
words, we don't model neither inheritances nor stacking, since this is
too complex for our needs. While a ``vgreduce`` operation on a LVM VG
could actually remove a PV from it, this will not be handled at the
framework level, but at individual operation level. The goal is that
this is a lightweight framework, for abstracting the different storage
operation, and not for modelling the storage hierarchy.


Locking improvements
~~~~~~~~~~~~~~~~~~~~

Current State and shortcomings
++++++++++++++++++++++++++++++

The class ``LockSet`` (see ``lib/locking.py``) is a container for one or many
``SharedLock`` instances. It provides an interface to add/remove locks and to
acquire and subsequently release any number of those locks contained in it.

Locks in a ``LockSet`` are always acquired in alphabetic order. Due to the way
we're using locks for nodes and instances (the single cluster lock isn't
affected by this issue) this can lead to long delays when acquiring locks if
another operation tries to acquire multiple locks but has to wait for yet
another operation.

In the following demonstration we assume to have the instance locks ``inst1``,
``inst2``, ``inst3`` and ``inst4``.

#. Operation A grabs lock for instance ``inst4``.
#. Operation B wants to acquire all instance locks in alphabetic order, but it
   has to wait for ``inst4``.
#. Operation C tries to lock ``inst1``, but it has to wait until
   Operation B (which is trying to acquire all locks) releases the lock again.
#. Operation A finishes and releases lock on ``inst4``. Operation B can
   continue and eventually releases all locks.
#. Operation C can get ``inst1`` lock and finishes.

Technically there's no need for Operation C to wait for Operation A, and
subsequently Operation B, to finish. Operation B can't continue until
Operation A is done (it has to wait for ``inst4``), anyway.

Proposed changes
++++++++++++++++

Non-blocking lock acquiring
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Acquiring locks for OpCode execution is always done in blocking mode. They
won't return until the lock has successfully been acquired (or an error
occurred, although we won't cover that case here).

``SharedLock`` and ``LockSet`` must be able to be acquired in a
non-blocking way. They must support a timeout and abort trying to acquire
the lock(s) after the specified amount of time.

Retry acquiring locks
^^^^^^^^^^^^^^^^^^^^^

To prevent other operations from waiting for a long time, such as described in
the demonstration before, ``LockSet`` must not keep locks for a prolonged period
of time when trying to acquire two or more locks. Instead it should, with an
increasing timeout for acquiring all locks, release all locks again and
sleep some time if it fails to acquire all requested locks.

A good timeout value needs to be determined. In any case should ``LockSet``
proceed to acquire locks in blocking mode after a few (unsuccessful) attempts
to acquire all requested locks.

One proposal for the timeout is to use ``2**tries`` seconds, where ``tries``
is the number of unsuccessful tries.

In the demonstration before this would allow Operation C to continue after
Operation B unsuccessfully tried to acquire all locks and released all
acquired locks (``inst1``, ``inst2`` and ``inst3``) again.

Other solutions discussed
+++++++++++++++++++++++++

There was also some discussion on going one step further and extend the job
queue (see ``lib/jqueue.py``) to select the next task for a worker depending on
whether it can acquire the necessary locks. While this may reduce the number of
necessary worker threads and/or increase throughput on large clusters with many
jobs, it also brings many potential problems, such as contention and increased
memory usage, with it. As this would be an extension of the changes proposed
before it could be implemented at a later point in time, but we decided to stay
with the simpler solution for now.


Feature changes
---------------

Ganeti Confd
~~~~~~~~~~~~

Current State and shortcomings
++++++++++++++++++++++++++++++
In Ganeti 2.0 all nodes are equal, but some are more equal than others. In
particular they are divided between "master", "master candidates" and "normal".
(Moreover they can be offline or drained, but this is not important for the
current discussion). In general the whole configuration is only replicated to
master candidates, and some partial information is spread to all nodes via
ssconf.

This change was done so that the most frequent Ganeti operations didn't need to
contact all nodes, and so clusters could become bigger. If we want more
information to be available on all nodes, we need to add more ssconf values,
which is counter-balancing the change, or to talk with the master node, which
is not designed to happen now, and requires its availability.
{'fraction_non_alphanumeric': 0.08591707134852447, 'fraction_numerical': 0.005416511019798282, 'mean_word_length': 4.441565040650406, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3903238', 'n_tokens_mistral': 2708, 'n_tokens_neox': 2534, 'n_words': 1610}