<commit_before>..
      Copyright 2015 Hewlett-Packard Development Company, L.P.
      All Rights Reserved.

      Licensed under the Apache License, Version 2.0 (the "License"); you may
      not use this file except in compliance with the License. You may obtain
      a copy of the License at

          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
      License for the specific language governing permissions and limitations
      under the License.

.. _searchlight-plugins:

Searchlight Plugin Documentation
================================

The search service determines the types of information that is searchable
via a plugin mechanism.

Installing Plugins
------------------

Plugins must be registered in ``setup.cfg``.

Within ``setup.cfg`` the setting within ``[entry_points]`` named
``searchlight.index_backend`` should list the plugin for each available
indexable type. After making a change, it's necessary to re-install the
python package (for instance with ``pip install -e .``).

Each plugin registered in ``setup.cfg`` is enabled by default. Typically it
should only be necessary to modify ``setup.cfg`` if you are installing a new
plugin. It is not necessary to modify ``[entry_points]`` to temporarily
enable or disable installed plugins. Once they are installed, they can be
disabled, enabled and configured in the ``searchlight-api.conf`` file.

Configuring Plugins
-------------------

After installation, plugins are configured in ``searchlight-api.conf``.

Default Plugin Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

All plugins inherit their configuration from a default configuration group of
``[resource_plugin:default]`` in ``searchlight-api.conf``. For example::

    [resource_plugin:default]
    enabled = true
    index_name = searchlight

The below table shows the available default configuration options. It is only
necessary to specify a configuration option in ``searchlight-api.conf`` if
you want to use a value other than the default value specified below.

+--------------------+---------------+-------------------------------------+---------------------------+
| Option             | Default value | Description                         | Action(s) Required        |
+====================+===============+=====================================+===========================+
| enabled            | true          | An installed plugin may be enabled  | | Restart services        |
|                    |               | or disabled. When disabled, it will | | Re-index affected types |
|                    |               | not be available for bulk indexing, |                           |
|                    |               | notification listening, or          |                           |
|                    |               | searching.                          |                           |
+--------------------+---------------+-------------------------------------+---------------------------+
| index_name         | searchlight   | The ElasticSearch index where the   | | Restart services        |
|                    |               | plugin resource documents will be   | | Re-index affected types |
|                    |               | stored in. It is recommended to not |                           |
|                    |               | change this unlesss needed.         |                           |
+--------------------+---------------+-------------------------------------+---------------------------+

.. note::

    After making changes to ``searchlight-api.conf`` you must perform the
    actions indicated in the table. Action notes:

    1. ``Restart services``: Restart all running ``searchlight-api`` *and*
       ``searchlight-listener`` processes.

    2. ``Re-index affected types``: You will need to re-index any resource
       types affected by the change. (See :doc:`indexingservice`).

Individual Plugin Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<commit_msg>Document unsearchable_fields configuration option<commit_after>..
      Copyright 2015 Hewlett-Packard Development Company, L.P.
      All Rights Reserved.

      Licensed under the Apache License, Version 2.0 (the "License"); you may
      not use this file except in compliance with the License. You may obtain
      a copy of the License at

          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
      License for the specific language governing permissions and limitations
      under the License.

.. _searchlight-plugins:

Searchlight Plugin Documentation
================================

The search service determines the types of information that is searchable
via a plugin mechanism.

Installing Plugins
------------------

Plugins must be registered in ``setup.cfg``.

Within ``setup.cfg`` the setting within ``[entry_points]`` named
``searchlight.index_backend`` should list the plugin for each available
indexable type. After making a change, it's necessary to re-install the
python package (for instance with ``pip install -e .``).

Each plugin registered in ``setup.cfg`` is enabled by default. Typically it
should only be necessary to modify ``setup.cfg`` if you are installing a new
plugin. It is not necessary to modify ``[entry_points]`` to temporarily
enable or disable installed plugins. Once they are installed, they can be
disabled, enabled and configured in the ``searchlight-api.conf`` file.

Configuring Plugins
-------------------

After installation, plugins are configured in ``searchlight-api.conf``.

.. note::

    After making changes to ``searchlight-api.conf`` you must perform the
    actions indicated in the tables below.

    1. ``Restart services``: Restart all running ``searchlight-api`` *and*
       ``searchlight-listener`` processes.

    2. ``Re-index affected types``: You will need to re-index any resource
       types affected by the change. (See :doc:`indexingservice`).

.. note::

    Unless you are changing to a non-default value, you do not need to
    specify any of the following configuration options.

Common Plugin Configuration Options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

There are common configuration options that all plugins honor. They are split
between inheritable configuration options and those that must always be
specified individually.

**Inheritable Common Configuration Options**

Some configuration options may be specified in a default configuration group
of ``[resource_plugin]`` in ``searchlight-api.conf`` and optionally overridden
in a specific plugin's configuration.

For example::

    [resource_plugin]
    index_name = searchlight

Inheritable options are listed below:

+---------------------+---------------+-------------------------------------+---------------------------+
| Option              | Default value | Description                         | Action(s) Required        |
+=====================+===============+=====================================+===========================+
| index_name          | searchlight   | The ElasticSearch index where the   | | Restart services        |
|                     |               | plugin resource documents will be   | | Re-index affected types |
|                     |               | stored in. It is recommended to not |                           |
|                     |               | change this unless needed.          |                           |
+---------------------+---------------+-------------------------------------+---------------------------+

**Non-Inheritable Common Configuration Options**

There are other common configuration options that all plugins honor, but must
be specified directly in that plugin's configuration group. They are
are not inherited from the ``[resource_plugin]`` configuration group.

See :ref:`individual-plugin-configuration` for more information and examples
on individual plugin configuration.

Non-inheritable options are listed below:

+---------------------+---------------+-------------------------------------+---------------------------+
| Option              | Default value | Description                         | Action(s) Required        |
+=====================+===============+=====================================+===========================+
| enabled             | true          | An installed plugin may be enabled  | | Restart services        |
|                     |               | or disabled. When disabled, it will | | Re-index affected types |
|                     |               | not be available for bulk indexing, |                           |
|                     |               | notification listening, or          |                           |
|                     |               | searching.                          |                           |
+---------------------+---------------+-------------------------------------+---------------------------+
| unsearchable_fields | <none>        | A comma separated list of fields    |                           |
|                     |               | (wildcards allowed) that are stored |                           |
|                     |               | in ElasticSearch, but can not be    |                           |
|                     |               | searched on regardless of the       |                           |
|                     |               | user's role (admin or not).         |                           |
|                     |               | The fields will still be returned   |                           |
|                     |               | in search results for admin users,  |                           |
|                     |               | but not normal users. These fields  |                           |
|                     |               | are typically specified for search  |                           |
|                     |               | performance, search accuracy,       |                           |
|                     |               | or security reasons.                |                           |
|                     |               | If a plugin has a hard-coded        |                           |
|                     |               | mapping for a specific field, it    |                           |
|                     |               | will take precedence over this      |                           |
|                     |               | configuration option.               |                           |
+---------------------+---------------+-------------------------------------+---------------------------+

.. _individual-plugin-configuration:

Individual Plugin Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
{'fraction_non_alphanumeric': 0.18456406095032007, 'fraction_numerical': 0.0018032639076728879, 'mean_word_length': 1.493705035971223, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 50, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5142298', 'n_tokens_mistral': 2282, 'n_tokens_neox': 2155, 'n_words': 926}