<commit_before>***************************
Administrator Documentation
***************************

When databroker is imported, it discovers catalogs available on the system.
User can list the discovered catalogs by importing a special global ``catalog``
object and listing its entries.

.. code:: python

   from databroker import catalog
   list(catalog)

This is the easiest way to users to discover what catalogs are available.
Catalogs can be provided in one of three ways:

1. Old-style "databroker v0.x" configuration files, for backward-compatibility
2. Intake-style catalog YAML files
3. Python packages that advertise catalogs via the ``intake.catalogs``
   entrypoint

Old-style databroker config
===========================

DataBroker v0.x used a custom YAML-based configuration file structure. See
:ref:`v0_configuration`. For backward-compatibility, configuration files
specifying MongoDB storage will be discovered and included in
``databroker.catalog``.

Migrating sqlite or HDF5 storage
--------------------------------

``databroker.v0`` interfaces with storage in MongoDB, sqlite, and HDF5.
``databroker.v1`` and ``databroker.v2`` drop support for sqlite and HDF5 and
add support for JSONL (newline-delimited JSON) and msgpack. For binary
file-based storage, we recommend using msgpack. Data can be migrated from
sqlite or HDF5 to msgpack like so:

.. code-block:: python

   import databroker.v0
   import suitcase.msgpack

   old = databroker.v0.Broker.named(YOUR_BROKER_NAMED)
   # Loop through every run in the old Broker.
   for run in old():
       # Load all the documents out of this run from their existing format and
       # write them into one file located at
       # `<DESTINATION_DIRECTORY>/<uid>.msgpack`.
       suitcase.msgpack.export(run.documents(), DESTINATION_DIRECTORY)

Create an intake catalog file somewhere in intake's search path, either at a
system ("global") location or a user location. You can list these locations
using the convenience function :func:`catalog_search_path`.

.. code:: python
<commit_msg>Simplify migration code, relying on v0 fallback.<commit_after>***************************
Administrator Documentation
***************************

When databroker is imported, it discovers catalogs available on the system.
User can list the discovered catalogs by importing a special global ``catalog``
object and listing its entries.

.. code:: python

   from databroker import catalog
   list(catalog)

This is the easiest way to users to discover what catalogs are available.
Catalogs can be provided in one of three ways:

1. Old-style "databroker v0.x" configuration files, for backward-compatibility
2. Intake-style catalog YAML files
3. Python packages that advertise catalogs via the ``intake.catalogs``
   entrypoint

Old-style databroker config
===========================

DataBroker v0.x used a custom YAML-based configuration file structure. See
:ref:`v0_configuration`. For backward-compatibility, configuration files
specifying MongoDB storage will be discovered and included in
``databroker.catalog``.

Migrating sqlite or HDF5 storage
--------------------------------

``databroker.v0`` interfaces with storage in MongoDB, sqlite, and HDF5.
``databroker.v1`` and ``databroker.v2`` drop support for sqlite and HDF5 and
add support for JSONL (newline-delimited JSON) and msgpack. For binary
file-based storage, we recommend using msgpack. Data can be migrated from
sqlite or HDF5 to msgpack like so:

.. code-block:: python

   from databroker import Broker
   import suitcase.msgpack

   # If the config file associated with YOUR_BROKER_NAME specifies sqlite or
   # HDF5 storage, then this will return a databroker.v0.Broker instance.
   db = Broker.named(YOUR_BROKER_NAME)
   # Loop through every run in the old Broker.
   for run in ():
       # Load all the documents out of this run from their existing format and
       # write them into one file located at
       # `<DESTINATION_DIRECTORY>/<uid>.msgpack`.
       suitcase.msgpack.export(run.documents(), DESTINATION_DIRECTORY)

Create an intake catalog file somewhere in intake's search path, either at a
system ("global") location or a user location. You can list these locations
using the convenience function :func:`catalog_search_path`.

.. code:: python

{'fraction_non_alphanumeric': 0.11439888164026095, 'fraction_numerical': 0.007222739981360671, 'mean_word_length': 4.191051995163241, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5612781', 'n_tokens_mistral': 1209, 'n_tokens_neox': 1134, 'n_words': 502}