<commit_before>============
Architecture
============


Math equation

.. math::

   (a + b)^2 = a^2 + 2ab + b^2013
   (a - b)^2 = a^2 - 2ab + b^2013

Second equation

.. math::
   \begin{cases}
   x_1 \gt 0, \quad \text{OR} &x_2 \gt 0 \\[2ex]
   x_1 + x_2 = 300
   \end{cases}


Math equation end<commit_msg>Add sections and fill in Architecture and Parser sections<commit_after>============
Architecture
============
Problem description
-------------------
User may have a variety of bare-metal node configuration, with different amount of disks, types of disks and their sizes, there should be a way to store best practises on what is the best way to do partitioning, so they can be applied for the most configuration cases without asking the end user to manually adjust the configuration of partitioning, with posibility to do that, if user wants to.

History
-------
First (and second) attempts to solve the problem has begun during development of `Fuel <https://wiki.openstack.org/wiki/Fuel>`_ project, special module `VolumeManager <https://github.com/openstack/fuel-web/blob/7.0/nailgun/nailgun/extensions/volume_manager/manager.py>`_ was created to solve the problem, it consumes `hardware information <https://github.com/openstack/fuel-web/blob/7.0/nailgun/nailgun/fixtures/sample_environment.json#L195-L232>`_ and `partitioning schema <https://github.com/openstack/fuel-web/blob/7.0/nailgun/nailgun/fixtures/openstack.yaml#L444-L577>`_, as result it generates sizes of spaces which should be allocated on the disks.

Current solution has `plenty of problems <https://blueprints.launchpad.net/bareon/+spec/dynamic-allocation>`_, it's hard and expensive to solve this problem in terms of old VolumeManager, because trivial algorithms and schema format don't allow us to extend it easily, handle all of the cases combined is not a trivial task to do if we try to solve the problem using brute-force.

List of terms
-------------
* **Disk** - a place where space can be allocated
* **Space** - an entity which can be allocated on several disks at once, a good example of a space is a `logical volume <https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)>`_ for lvm, another one is partition
* **Dynamic schema** - a schema without specific sizes, it's a schema which is used by user to specify partitioning schema without details
* **Static schema** - a schema for `Bareon <https://wiki.openstack.org/wiki/Bareon>`_ which requires exact space <-> disk mapping with exact size of each space

High level architecture
-----------------------

::
    +-------------------------+
    |                         |
    |  Dynamic schema parser  |
    |                         |
    +------------+------------+
                 |
                 |
    +------------v------------+
    |                         |
    |    Allocation solver    |
    |                         |
    +------------+------------+
                 |
                 |
    +------------v-------------+
    |                          |
    |    Solution convertor    |
    |                          |
    +--------------------------+

* **Dynamic schema parser** - parses an input from the user and prepares the data which can be consumed by Allocation solver
* **Allocation solver** - an algorithm which takes dynamic schema and produces a static schema
* **Solution convertor** - a result which is produced by solver, should be parsed and converted into `Bareon <https://wiki.openstack.org/wiki/Bareon>`_ consumable format, for example for `Logical Volume <https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)>`_ Solution convertor should generate a physical volume for each disk, where it's allocated

Dynamic schema parser
---------------------

In the current version we user flat schema, it's a list which consists dictionaries.

Basic syntax
~~~~~~~~~~~~

* **id** - id of a space
* **type** - type of a space, for example Volume Group or Logical Volume
* **max_size** - maximum size which is allowed for the space
* **min_size** - minimal size which is allowed for the space
* **size** - a static size, it's similar as to set for **min_size** and **max_size** the same value
* **contains** - is required for hierarchical spaces such as Volume Group

Also there are couple of different attributes, such as **mount**, **fs_type**, which are self-explanatory. A list of such attributes is not complete and may be easily extended in the future.

.. code-block:: yaml

    - id: os
      type: vg
      contains:
        - id: swap
        - id: root

    - id: root
      type: lv
      max_size: 10000
      min_size: 5000
      mount: /
      fs_type: ext4

    - id: swap
      type: lv
      size: 2000
      fs_type: swap


Dynamic parameters
~~~~~~~~~~~~~~~~~~

What if user wants to allocate a size of space based on some different parameter?
As an example lets consider a size of swap which has to be based on amount of RAM the node has.

.. code-block:: yaml

    ram: 4096
    disks:
      - id: /dev/disk/by-id/id-for-sda
        path: /dev/disk/by-path/path-for-sda
        dev: /dev/sda
        type: hdd
        vendor: Hitachi
        size: 5000

From Hardware Information example we can see that the node has 4096 megabytes of RAM, according to `best practises <https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s2-diskpartrecommend-ppc.html>`_ on swap size allocation swap size has to be twice bigger than current RAM.

.. code-block:: yaml

    - id: swap
      type: lv
      size: 2000
      fs_type: swap
      size: |
        yaql=let(ram => $.get(ram, 1024)) ->
        selectCase(
          $ram <= 2048,
          $ram > 2048 and $ram < 8192,
          $ram > 8192 and $ram < 65536).
        switchCase(
          $ram * 2,
          $ram,
          $ram / 2,
          4096)

In order to implement an algorithm of swap size calculation we use `<YAQL https://github.com/openstack/yaql>`_, which is small but powerful enough query language. Any value of the parameter which matches to **yaql=yaql expression** will be evaluated using YAQL, execution result will be passed as is to the Solver.

Solver
------

Problem description
~~~~~~~~~~~~~~~~~~~

Solver description
~~~~~~~~~~~~~~~~~~

Two volumes
~~~~~~~~~~~

Two disks
~~~~~~~~~

Minimal size
~~~~~~~~~~~~

Maximum size
~~~~~~~~~~~~

Integer solution
~~~~~~~~~~~~~~~~

Ordering
~~~~~~~~

Weight
~~~~~~

Best with disks
~~~~~~~~~~~~~~~

Advanced ordering
~~~~~~~~~~~~~~~~~
Describe sequence


{'fraction_non_alphanumeric': 0.14891003991403132, 'fraction_numerical': 0.016119128031931225, 'mean_word_length': 2.504572350726197, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 19, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7191462', 'n_tokens_mistral': 1889, 'n_tokens_neox': 1727, 'n_words': 717}