<commit_before>=============
Configuration
=============

DXR learns how to index and serve your source trees by means of an ini-formatted
configuration file:

.. include:: example-configuration.rst

When you invoke :program:`dxr index`, it defaults to reading :file:`dxr.config`
in the current directory::

    dxr index

Or you can pass in a config file explicitly::

    dxr index --config /some/place/dxr.config


Sections
========

The configuration file is divided into sections. The ``[DXR]`` section holds
global options; each other section describes a tree to be indexed.

You can use all the fancy interpolation features of Python's
`ConfigParser <http://docs.python.org/library/configparser.html>`__ class to
save repetition.

[DXR] Section
-------------

Here are the options that can live in the ``[DXR]`` section. For options
representing path names, relative paths are relative to the directory
containing the config file.

``temp_folder``
    A ``format()``-style template for deciding where to store temporary files
    used while indexing. The token ``{tree}`` will be replaced with the name
    of each tree you index. Default: ``dxr-temp-{tree}``. It's a good idea to
    keep this out of :file:`/tmp` if it's on a small partition, since it can
    grow to tens of gigabytes on a large codebase.

``log_folder``
    A ``format()``-style template for deciding where to store log files
    written while indexing. The token ``{tree}`` will be replaced with the name
    of the tree being indexed. Default: ``dxr-logs-{tree}`` (in the current
    working directory).

``disabled_plugins``
    Names of plugins to disable. Default: empty

``enabled_plugins``
    Names of plugins to enable. Default: ``*``

``generated_date``
    The "generated on" date stamped at the bottom of every DXR web page, in
    RFC-822 (also known as RFC 2822) format. Default: the time the indexing run
    started

``workers``
    Number of concurrent processes to use for building and indexing projects.
    Default: the number of CPUs on the system. Set to 0 to use no worker
    processes and do everything in the master process. This is handy for
    debugging.

``skip_stages``
    Build/indexing stages to skip, for debugging: ``build``, ``index``, or
    both, whitespace-separated. Default: none

``es_alias``
    A ``format()``-style template for coming up with elasticsearch alias
    names. These live in the same namespace as indices, so don't pave over any
    index name you're already using. The variables ``{format}`` and ``{tree}``
    will be substituted, and their meanings are as in ``es_index``. Default:
    ``dxr_{format}_{tree}``.

``es_index``
    A ``format()``-style template for coming up with elasticsearch index
    names. The variable ``{tree}`` will be replaced with the tree name,
    ``{format}`` will be replaced with the format version, and ``{unique}``
    will be replaced with a unique ID to keep a tree's new index from
    colliding with the old one. The unique ID includes a random number and the
    build hosts's MAC address so errant concurrent builds on different hosts
    at least won't clobber each other. Default: ``dxr_{format}_{tree}_{unique}``

``es_catalog_replicas``
    The number of elasticsearch replicas to make of the :term:`catalog index`.
    This is read often and written only when an indexing run completes, so
    crank it up so there's a replica on every node for best performance. But
    remember that writes will hang if at least half of the attempted copies
    aren't available. Default: ``1``

``es_indexing_timeout``
    The number of seconds DXR should wait for elasticsearch responses during
    indexing. Default: 60

``es_refresh_interval``
    The number of seconds between elasticsearch's consolidation passes during
    indexing. Set to -1 to do no refreshes at all, except directly after an
    indexing run completes. Default: 60

Web App Options That Need a Restart
```````````````````````````````````

These options are used by the DXR web app (though some are used at index time
as well). They are not frozen into the :term:`catalog index` but rather are
read when the web app starts up. Thus, the web app must be restarted to see
new values of these.

``default_tree``
    The tree to redirect to when you visit the root of the site. Default: the
    first tree in the config file
<commit_msg>Alphabetize config options.<commit_after>=============
Configuration
=============

DXR learns how to index and serve your source trees by means of an ini-formatted
configuration file:

.. include:: example-configuration.rst

When you invoke :program:`dxr index`, it defaults to reading :file:`dxr.config`
in the current directory::

    dxr index

Or you can pass in a config file explicitly::

    dxr index --config /some/place/dxr.config


Sections
========

The configuration file is divided into sections. The ``[DXR]`` section holds
global options; each other section describes a tree to be indexed.

You can use all the fancy interpolation features of Python's
`ConfigParser <http://docs.python.org/library/configparser.html>`__ class to
save repetition.

[DXR] Section
-------------

Here are the options that can live in the ``[DXR]`` section. For options
representing path names, relative paths are relative to the directory
containing the config file.

``disabled_plugins``
    Names of plugins to disable. Default: empty

``enabled_plugins``
    Names of plugins to enable. Default: ``*``

``es_alias``
    A ``format()``-style template for coming up with elasticsearch alias
    names. These live in the same namespace as indices, so don't pave over any
    index name you're already using. The variables ``{format}`` and ``{tree}``
    will be substituted, and their meanings are as in ``es_index``. Default:
    ``dxr_{format}_{tree}``.

``es_index``
    A ``format()``-style template for coming up with elasticsearch index
    names. The variable ``{tree}`` will be replaced with the tree name,
    ``{format}`` will be replaced with the format version, and ``{unique}``
    will be replaced with a unique ID to keep a tree's new index from
    colliding with the old one. The unique ID includes a random number and the
    build hosts's MAC address so errant concurrent builds on different hosts
    at least won't clobber each other. Default: ``dxr_{format}_{tree}_{unique}``

``es_catalog_replicas``
    The number of elasticsearch replicas to make of the :term:`catalog index`.
    This is read often and written only when an indexing run completes, so
    crank it up so there's a replica on every node for best performance. But
    remember that writes will hang if at least half of the attempted copies
    aren't available. Default: ``1``

``es_indexing_timeout``
    The number of seconds DXR should wait for elasticsearch responses during
    indexing. Default: 60

``es_refresh_interval``
    The number of seconds between elasticsearch's consolidation passes during
    indexing. Set to -1 to do no refreshes at all, except directly after an
    indexing run completes. Default: 60

``generated_date``
    The "generated on" date stamped at the bottom of every DXR web page, in
    RFC-822 (also known as RFC 2822) format. Default: the time the indexing run
    started

``log_folder``
    A ``format()``-style template for deciding where to store log files
    written while indexing. The token ``{tree}`` will be replaced with the name
    of the tree being indexed. Default: ``dxr-logs-{tree}`` (in the current
    working directory).

``skip_stages``
    Build/indexing stages to skip, for debugging: ``build``, ``index``, or
    both, whitespace-separated. Default: none

``temp_folder``
    A ``format()``-style template for deciding where to store temporary files
    used while indexing. The token ``{tree}`` will be replaced with the name
    of each tree you index. Default: ``dxr-temp-{tree}``. It's a good idea to
    keep this out of :file:`/tmp` if it's on a small partition, since it can
    grow to tens of gigabytes on a large codebase.

``workers``
    Number of concurrent processes to use for building and indexing projects.
    Default: the number of CPUs on the system. Set to 0 to use no worker
    processes and do everything in the master process. This is handy for
    debugging.

Web App Options That Need a Restart
```````````````````````````````````

These options are used by the DXR web app (though some are used at index time
as well). They are not frozen into the :term:`catalog index` but rather are
read when the web app starts up. Thus, the web app must be restarted to see
new values of these.

``default_tree``
    The tree to redirect to when you visit the root of the site. Default: the
    first tree in the config file

{'fraction_non_alphanumeric': 0.09548254620123203, 'fraction_numerical': 0.0031941592516541184, 'mean_word_length': 3.806469298245614, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4985040', 'n_tokens_mistral': 2417, 'n_tokens_neox': 2292, 'n_words': 1224}