<commit_before>Retrieving metadata, tabular data, and image data
*************************************************

Problem
=======

Retrieve metadata, tabular data, or image data for analysis, processing, or
export.

Approach
========

Use the databroker, the all-in-one interface to saved data.

Search for the *header(s)* of the run(s) of interest. Explore the "raw"
interface, which provides maximum flexiblity and performance. Then use
some "convenience functions" for loading a table or a stack of images.

Example Solution
================

First, serach for the *header(s)* of the run(s) of interest. (The headers
contain metadata. See :doc:`/examples/header-contents` for details.)

.. ipython:: pyton
    :suppress:

    from bluesky.scans import scan
    from bluesky import RunEngine
    # generate fake data


.. ipython:: python

    db.get_table(h)

    db.get_images(h, 'img')<commit_msg>Flesh of 'retrieving data' example.<commit_after>Retrieving metadata, tabular data, and image data
*************************************************

Problem
=======

Retrieve metadata, tabular data, or image data for analysis, processing, or
export.

Approach
========

Use the databroker, the all-in-one interface to saved data.

Retrieve the metadata for the run(s) of interest. Retrieve the data itself
in three different modes:

* a general-purpose method, which provides maximum flexiblity and performance
* a convenient method for retrieving tabular data
* a convenient method for retrieving image data

Example Solution
================

.. ipython:: python
    :suppress:

    from bluesky import RunEngine
    from bluesky.plans import scan
    from bluesky.examples import motor, det
    import metadatastore.commands as mds
    # import filestore.api as fs
    # from databroker import Broker
    # db = Broker(mds, fs)
    from databroker import db
    RE = RunEngine({})
    RE.subscribe_lossless('all', mds.insert)

We'll preface this example by generating some example data.

.. ipython:: python

    uid, = RE(scan([det], motor, -10, 10, 15))

The unique id of the data set has been stashed in the variable ``uid``. We can
use that to retrieve the data from the databroker.

.. ipython:: python

    h = db[uid]

What we get back is a *header*, which contains all of the metadata from the
run.  For example, we can review the names of the detector(s) involved:

.. ipython:: python

    h['start']['detectors']

There is a lot of information in ``h``. See :doc:`/examples/header-contents`.
What about the data itself?

General-Purpose Method
----------------------

.. ipython:: python

    events = db.get_events(h)

In the variable ``events``, we now have a collection of documents
(dictionary-like mappings of names to values). Each event corresponds to 
a single data point, a row in table.

For performance reasons, the data has not actually been loaded yet. The data
is loaded one point at a time if loop through ``events``. (This is very
useful for applications where we don't need to load the entire data set.)

To load the entire data set once, convert ``events`` to a list.

.. ipython:: python

    events = list(events)  # for large data sets, this takes awhile

Let's look at all the data in the events.

.. ipython:: python

    [event['data'] for event in events]

You might be thinking, "Just give me data!" As promised, the general-purpose
method is flexible, but it lacks terseness. For more direct methods, read on!

To learn more about the structure of an ``event``, refer to the
`overview of the document model <nsls-ii.github.io/architecture-overview.html>`_.

Retrieving a Table
------------------

.. ipython:: python

    db.get_table(h)

The result is a DataFrame. One can access individual columns like so:

.. ipython:: python

    table = db.get_table(h)
    table['det']

perform fast array computations using numpy

.. ipython:: python

    import numpy as np

    np.mean(table)

and much, much more.

.. note::

    The variable ``table`` here is a pandas DataFrame, scientific Python's
    answer to the spreadsheet. Read the
    `pandas documentation <http://pandas.pydata.org/pandas-docs/stable/>`_
    for more. It's an extremely powerful package for analyzing tabular
    data.

Narrowing the Results
+++++++++++++++++++++

The ``get_table`` method accepts several optional arguments which can be used
to filter the results (and corespondingly speed up the retrieval). Examples:

.. ipython:: python

    db.get_table(h, ['det'])  # just include the 'det' column

Retrieving Images
-----------------

.. warning::

    The short answer is, ``db.get_table(h, 'image_field_name')``. This section
    is to be written.
{'fraction_non_alphanumeric': 0.13029525032092426, 'fraction_numerical': 0.0012836970474967907, 'mean_word_length': 3.6517412935323383, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1669477', 'n_tokens_mistral': 1369, 'n_tokens_neox': 1309, 'n_words': 583}