<commit_before>Muninn extension developer documentation
========================================

This document is intended for muninn extension developers. Muninn is a generic
archiving framework. To be able to use it to archive specific (types of)
products, it is necessary to install (or implement) one or more extensions.

Readers of this document are assumed to be familiar with the content of the
muninn README.rst file, in particular the sections "Extensions", "Data types",
"Namespaces", and "Links".

A muninn extension is a Python module or package that implements the muninn
extension interface. Muninn defines three types of extensions: namespace
extensions (that contain namespace definitions), product type extensions
(that contain product type plug-ins) and remote backend extensions (that
contain remote backend plug-ins).

A namespace is a set of related properties, i.e. a set of (key, value) pairs.
The namespace definition specifies the keys (field names) available within the
namespace, their type, and whether or not they are optional.

For example, this is the definition of the ``core`` namespace of muninn (see
also the file ``core.py`` included in the muninn source distribution): ::

  from muninn.schema import *

  class Core(Mapping):
    uuid = UUID()
    active = Boolean(index=True)
    hash = Text(optional=True, index=True)
    size = Long(optional=True, index=True)
    metadata_date = Timestamp(index=True)
    archive_date = Timestamp(optional=True, index=True)
    archive_path = ArchivePath(optional=True)
    product_type = Text(index=True)
    product_name = Text(index=True)
    physical_name = Basename(index=True)
    validity_start = Timestamp(optional=True, index=True)
    validity_stop = Timestamp(optional=True, index=True)
    creation_date = Timestamp(optional=True, index=True)
    footprint = Geometry(optional=True)
    remote_url = Remote(optional=True)

By default, properties are required. As can be seen in the example, this can be changed by specifying ``optional=True``. By default, properties are also not indexed in the database backend. This can be changed by specifying ``index=True``.

A product type plug-in is an instance of a class that handles all product type
specific details. The most important function of a product type plug-in is to
extract properties from a product and return them in a form the archiving
framework understands.

To represent product properties, a class called ``muninn.Struct`` is used,
which is essentially an empty class derived from object. Product properties are
added to this class via injection. Think of it as a dictionary, except that you
can also use ``.`` to access the value bound to a specific product property.
A ``muninn.Struct`` can be initialized with a python dictionary. This will also
convert all members that are dictionaries into ``muninn.Struct`` objects.

By convention, product properties are named <namespace name>.<property name>.
This means you usually have a single top-level Struct instance, that contains a
separate Struct instance for each namespace. For example: ::

  from muninn import Struct

  properties = Struct()
  properties.core = Struct()
  properties.core.product_type = "ABCD"
  properties.core.creation_date = datetime.datetime.utcnow()
  ... more of the same ...

  properties.xml_pi = Struct()<commit_msg>Wrap text to 80 columns<commit_after>Muninn extension developer documentation
========================================

This document is intended for muninn extension developers. Muninn is a generic
archiving framework. To be able to use it to archive specific (types of)
products, it is necessary to install (or implement) one or more extensions.

Readers of this document are assumed to be familiar with the content of the
muninn README.rst file, in particular the sections "Extensions", "Data types",
"Namespaces", and "Links".

A muninn extension is a Python module or package that implements the muninn
extension interface. Muninn defines three types of extensions: namespace
extensions (that contain namespace definitions), product type extensions
(that contain product type plug-ins) and remote backend extensions (that
contain remote backend plug-ins).

A namespace is a set of related properties, i.e. a set of (key, value) pairs.
The namespace definition specifies the keys (field names) available within the
namespace, their type, and whether or not they are optional.

For example, this is the definition of the ``core`` namespace of muninn (see
also the file ``core.py`` included in the muninn source distribution): ::

  from muninn.schema import *

  class Core(Mapping):
    uuid = UUID()
    active = Boolean(index=True)
    hash = Text(optional=True, index=True)
    size = Long(optional=True, index=True)
    metadata_date = Timestamp(index=True)
    archive_date = Timestamp(optional=True, index=True)
    archive_path = ArchivePath(optional=True)
    product_type = Text(index=True)
    product_name = Text(index=True)
    physical_name = Basename(index=True)
    validity_start = Timestamp(optional=True, index=True)
    validity_stop = Timestamp(optional=True, index=True)
    creation_date = Timestamp(optional=True, index=True)
    footprint = Geometry(optional=True)
    remote_url = Remote(optional=True)

By default, properties are required. As can be seen in the example, this can be
changed by specifying ``optional=True``. By default, properties are also not
indexed in the database backend. This can be changed by specifying
``index=True``.

A product type plug-in is an instance of a class that handles all product type
specific details. The most important function of a product type plug-in is to
extract properties from a product and return them in a form the archiving
framework understands.

To represent product properties, a class called ``muninn.Struct`` is used,
which is essentially an empty class derived from object. Product properties are
added to this class via injection. Think of it as a dictionary, except that you
can also use ``.`` to access the value bound to a specific product property.
A ``muninn.Struct`` can be initialized with a python dictionary. This will also
convert all members that are dictionaries into ``muninn.Struct`` objects.

By convention, product properties are named <namespace name>.<property name>.
This means you usually have a single top-level Struct instance, that contains a
separate Struct instance for each namespace. For example: ::

  from muninn import Struct

  properties = Struct()
  properties.core = Struct()
  properties.core.product_type = "ABCD"
  properties.core.creation_date = datetime.datetime.utcnow()
  ... more of the same ...

  properties.xml_pi = Struct()
{'fraction_non_alphanumeric': 0.0808472553699284, 'fraction_numerical': 0.0002983293556085919, 'mean_word_length': 4.765262252794497, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '861748', 'n_tokens_mistral': 1696, 'n_tokens_neox': 1650, 'n_words': 825}