<commit_before>
Storage Plugin Developer's Guide for Intel® Manager for Lustre* Software
========================================================================

Introduction
------------

Storage plugins are responsible for delivering information about
entities that are not part of the Linux\*/Lustre\* stack.  This primarily means
storage controllers and network devices, but the plugin system is generic and 
does not limit the type of objects that can be reported.

To present device information to the Command Center, a Python\* module is written using
the Storage Plugin API described in this document:

* The objects to be reported are described by declaring a series of
  Python classes: :ref:`storage_resources`
* Certain of these objects are used to store the contact information
  such as IP addresses for managed devices: :ref:`scannable_storage_resources`
* A main plugin class is implemented to provide required hooks for
  initialization and teardown: :ref:`storage_plugins`

The API is designed to minimize the lines of code required to write a plugin, 
minimize the duplication of effort between different plugins, and make as much
of the plugin code as possible declarative in style to minimize the need for 
per-plugin testing.  For example, rather than having plugins procedurally
check for resources that are in bad states during an update, we provide the means to declare
conditions that are automatically checked.  Similarly, rather than requiring explicit 
notification from the plugin when a resource attribute is changed, we detect
assignments to attributes and schedule transmission of updates in the background.

Plugins can provide varying levels of functionality.  The most basic plugins can provide
only discovery of storage resources at the time the plugin is loaded (requiring a manual
restart to detect any changes).  Most plugins will provide at least the initial discovery
and live detection of added/removed resources of certain classes, for example LUNs
as they are created and destroyed.  On a per-resource-class basis, alert conditions
can be added to report issues with the resources such as pools being rebuilt or 
physical disks that fail.

In addition to reporting a set of resources, plugins report a set of relationships between
the resources, used for associating problems with one resource with another resource.  This 
takes the form of an "affects" relationship between resources, for example the health of 
a physical disk affects the health of its pool, and the health of a pool affects LUNs
in that pool.  These relationships allow the effects of issues to be traced all the 
way up to the Lustre file system level and an appropriate drill-down user interface to be provided.

Terminology
~~~~~~~~~~~

plugin
  A Python module providing a number of classes inheriting from the
  Plugin and Resource classes.

resource class
  A subclass of Resource declaring a means of identification, attributes,
  statistics and alert conditions.

resource
  An instance of a particular resource class, with a unique identifier.


.. _storage_resources:

Declaring Resources
-------------------

A `Resource` represents a single, uniquely identified object.  It may be a physical
device such as a physical hard drive, or a virtual object such as a storage pool or 
virtual machine.

.. code-block:: python

   from chroma_core.lib.storage_plugin.api import resources, identifiers, attributes, statistics

   class HardDrive(resources.Resource):
       serial_number = attributes.String()
       capacity = attributes.Bytes()
       temperature = statistics.Gauge(units = 'C')

       class Meta:
           identifier = identifiers.GlobalId('serial_number')

In general, storage resources may inherit from Resource directly, but
optionally they may inherit from a built-in resource class as a way of 
identifying common resource types for presentation purposes.  See 
:ref:`storage_plugin_builtin_resource_classes` for the available built-in
resource types.

Attributes
~~~~~~~~~~

The ``serial_number`` and ``capacity`` attributes of the HardDrive class are
from the ``attributes`` module.  These are special classes which:

* Apply validation conditions to the attributes
* Act as metadata for the presentation of the attribute in the Command Center user interface

Various attribute classes are available for use, see :ref:`storage_plugin_attribute_classes`.

Attributes may be optional or mandatory.  They are mandatory by default. To 
make an attribute optional, pass ``optional = True`` to the constructor.

Statistics
~~~~~~~~~~

The ``temperature`` attribute of the HardDrive class is an example of
a resource statistic.  Resource statistics differ from resource
attributes in the way they are presented to the user.  See :ref:`storage_plugin_statistic_classes`
for more on statistics.

Statistics are stored at runtime by assigning to the relevant
attribute of a storage resource instance.  For example, for a 
``HardDrive`` instance ``hd``, assigning ``hd.temperature = 20`` updates
the statistic.

Identifiers
~~~~~~~~~~~

Every Resource subclass is required to have an ``identifier`` attribute
that declares which attributes are to be used to uniquely identify the resource.

If a resource has a universally unique name and may be reported from more than one
place (for example, a physical disk which may be reported from more than one 
controller), use ``chroma_core.lib.storage_plugin.api.identifiers.GlobalId``.  For example, 
in the ``HardDrive`` class described above, each drive has a unique factory-assigned ID.

If a resource has an identifier that is scoped to a *scannable* storage resource or
that always belongs to a particular scannable storage resource, use
``chroma_core.lib.storage_plugin.api.identifiers.ScopedId``.

Either of the above classes is initialized with a list of attributes which
in combination are a unique identifier.  For example, if a true hard drive 
identifier is unavailable, a drive might be identified within a particular couplet 
by its shelf and slot number, like this:
::

    class HardDrive(Resource):
        class Meta:<commit_msg>Removed space.<commit_after>
Storage Plugin Developer's Guide for Intel® Manager for Lustre* Software
========================================================================

Introduction
------------

Storage plugins are responsible for delivering information about
entities that are not part of the Linux\*/Lustre\* stack.  This primarily means
storage controllers and network devices, but the plugin system is generic and 
does not limit the type of objects that can be reported.

To present device information to the Command Center, a Python\* module is written using
the Storage Plugin API described in this document:

* The objects to be reported are described by declaring a series of
  Python classes: :ref:`storage_resources`
* Certain of these objects are used to store the contact information
  such as IP addresses for managed devices: :ref:`scannable_storage_resources`
* A main plugin class is implemented to provide required hooks for
  initialization and teardown: :ref:`storage_plugins`

The API is designed to minimize the lines of code required to write a plugin, 
minimize the duplication of effort between different plugins, and make as much
of the plugin code as possible declarative in style to minimize the need for 
per-plugin testing.  For example, rather than having plugins procedurally
check for resources that are in bad states during an update, we provide the means to declare
conditions that are automatically checked.  Similarly, rather than requiring explicit 
notification from the plugin when a resource attribute is changed, we detect
assignments to attributes and schedule transmission of updates in the background.

Plugins can provide varying levels of functionality.  The most basic plugins can provide
only discovery of storage resources at the time the plugin is loaded (requiring a manual
restart to detect any changes).  Most plugins will provide at least the initial discovery
and live detection of added/removed resources of certain classes, for example LUNs
as they are created and destroyed.  On a per-resource-class basis, alert conditions
can be added to report issues with the resources such as pools being rebuilt or 
physical disks that fail.

In addition to reporting a set of resources, plugins report a set of relationships between
the resources, used for associating problems with one resource with another resource.  This 
takes the form of an "affects" relationship between resources, for example the health of 
a physical disk affects the health of its pool, and the health of a pool affects LUNs
in that pool.  These relationships allow the effects of issues to be traced all the 
way up to the Lustre file system level and an appropriate drill-down user interface to be provided.

Terminology
~~~~~~~~~~~

plugin
  A Python module providing a number of classes inheriting from the
  Plugin and Resource classes.

resource class
  A subclass of Resource declaring a means of identification, attributes,
  statistics and alert conditions.

resource
  An instance of a particular resource class, with a unique identifier.


.. _storage_resources:

Declaring Resources
-------------------

A `Resource` represents a single, uniquely identified object.  It may be a physical
device such as a physical hard drive, or a virtual object such as a storage pool or 
virtual machine.

.. code-block:: python

   from chroma_core.lib.storage_plugin.api import resources, identifiers, attributes, statistics

   class HardDrive(resources.Resource):
       serial_number = attributes.String()
       capacity = attributes.Bytes()
       temperature = statistics.Gauge(units = 'C')

       class Meta:
           identifier = identifiers.GlobalId('serial_number')

In general, storage resources may inherit from Resource directly, but
optionally they may inherit from a built-in resource class as a way of 
identifying common resource types for presentation purposes.  See 
:ref:`storage_plugin_builtin_resource_classes` for the available built-in
resource types.

Attributes
~~~~~~~~~~

The ``serial_number`` and ``capacity`` attributes of the HardDrive class are
from the ``attributes`` module.  These are special classes which:

* Apply validation conditions to the attributes
* Act as metadata for the presentation of the attribute in the Command Center user interface

Various attribute classes are available for use, see :ref:`storage_plugin_attribute_classes`.

Attributes may be optional or mandatory.  They are mandatory by default. To 
make an attribute optional, pass ``optional = True`` to the constructor.

Statistics
~~~~~~~~~~

The ``temperature`` attribute of the HardDrive class is an example of
a resource statistic. Resource statistics differ from resource
attributes in the way they are presented to the user.  See :ref:`storage_plugin_statistic_classes`
for more on statistics.

Statistics are stored at runtime by assigning to the relevant
attribute of a storage resource instance.  For example, for a 
``HardDrive`` instance ``hd``, assigning ``hd.temperature = 20`` updates
the statistic.

Identifiers
~~~~~~~~~~~

Every Resource subclass is required to have an ``identifier`` attribute
that declares which attributes are to be used to uniquely identify the resource.

If a resource has a universally unique name and may be reported from more than one
place (for example, a physical disk which may be reported from more than one 
controller), use ``chroma_core.lib.storage_plugin.api.identifiers.GlobalId``.  For example, 
in the ``HardDrive`` class described above, each drive has a unique factory-assigned ID.

If a resource has an identifier that is scoped to a *scannable* storage resource or
that always belongs to a particular scannable storage resource, use
``chroma_core.lib.storage_plugin.api.identifiers.ScopedId``.

Either of the above classes is initialized with a list of attributes which
in combination are a unique identifier.  For example, if a true hard drive 
identifier is unavailable, a drive might be identified within a particular couplet 
by its shelf and slot number, like this:
::

    class HardDrive(Resource):
        class Meta:
{'fraction_non_alphanumeric': 0.06289769946157611, 'fraction_numerical': 0.00032631750693424703, 'mean_word_length': 4.618240146654445, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '818405', 'n_tokens_mistral': 2840, 'n_tokens_neox': 2737, 'n_words': 1654}