<commit_before>Data handling with the Data Class
=================================

*Note: This module is still a work-in-progress and the usage of these classes and/or functions may change in the future.*

The following contains examples of how to use the various features in the :py:class:`.Data`, whose purpose is to simplify the handling of neutron scattering data and provide basic analysis and visualization functions.

Initialization: loading data
----------------------------
There are a couple of options to initialize the :py:class:`.Data` object. If your data is in SPICE (HFIR, ORNL), ICE or ICP (NCNR) formats then you can use :py:meth:`.Data.load_file`, otherwise you will need to load the data and pass it to :py:class:`.Data` manually.

Load from file
^^^^^^^^^^^^^^
First, lets assume that you have a data file, ``'scan.dat'`` from a HFIR instrument (SPICE format):

>>> from neutronpy.tools import Data
>>> data = Data()
>>> data.load_file('scan.dat', mode='SPICE')

This builds ``data.Q`` automatically and loads the following information from the file:

* detector counts
* monitor counts
* h, k, l
* energy transfer
* temperature
* ei, ef (future feature)
* time (future feature)

Currently, other data from the files is not included, but some additional support will be added in the future.

**Note: The primary problem is there is no standard variable name set between different file formats, and documentation for the file formats are not complete in some cases. Additionally, instruments at the same facility often use different software, so formats are frequently incompatible.**

If you want to load more than one file at a time, simple add the file names as separate arguments, *e.g.*

>>> data.load_file('scan1.dat', 'scan2.dat', mode='SPICE')

Pass pre-loaded data
^^^^^^^^^^^^^^^^^^^^
Assuming that your data is in a format that is not supported by :py:meth:`.Data.load_file` you will need to load the data yourself and pass it to the :py:class:`.Data` class and build ``Data.Q`` using :py:meth:`.Data.build_Q`. To build ``Data.Q`` you must have defined ``h``, ``k``, ``l``, ``e``, and ``temp``.

>>> data = Data(h=h, k=k, l=l, e=e, temp=temp, detector=detector, monitor=monitor, time=time)
>>> data.Q = data.build_Q()

``Data`` properties
---------------
Below I outline some of the most common properties that you will want of the :py:class:`.Data` class.

Intensity and error
^^^^^^^^^^^^^^^^^^^
Intensity, *i.e.* ``detector / monitor * m0`` and square-root error are respectively obtained by

>>> data.intensity()
>>> data.error()

Monitor normalization
^^^^^^^^^^^^^^^^^^^^^
If you want to normalize to a particular monitor ``m0`` then you will need to define it, *e.g.*

>>> data.m0 = 1e5

If you do not choose a ``m0``, when you call :py:meth:`.Data.intensity()` one will be defined for you based on the ``monitor`` already defined in ``data``.

The ``Q`` vector
^^^^^^^^^^^^^^^^
In this case, ``Q`` is collection of column arrays defined as ``[h, k, l, e, temp]``, with ``data.Q.shape = (N, 5)``. Typically, one would expect that ``temp`` not be included in ``Q``, but for the purposes of rebinning it is included currently. *In the future, rebinning may be expanded to include other arbitrary dimensions, rather than just these five.* If data has been loaded from one of the supported file formats, or :py:meth:`.Data.build_Q` has been used then these variables can also be accessed separately by:

>>> h = data.h
>>> k = data.k
>>> l = data.l
>>> e = data.e
>>> temp = data.temp

``Data`` operations
-------------------
Combining data is as easy as adding multiple ``Data`` objects together, *e.g.*

>>> data1 = Data()
>>> data1.load_file('scan1.dat', mode='SPICE')

>>> data2 = Data()
>>> data2.load_file('scan2.dat', mode='SPICE')

>>> data = data1 + data2

This will combine monitor and detector counts for existing points and concatenate unique points in the two objects to create a new ``data`` object.

Subtracting works in a similar way, but keep in mind that in its current form it doesn't interpolate, so if ``Q`` is different between the two ``data`` variables then you will end up with negative intensities at positions where there isn't an overlapping Q. Proper background subtraction will be implemented in the future.

The ``*``, ``/`` and ``**`` operators only act on the detector variable. This is useful for example if you want to apply the detailed balance factor obtained from :py:meth:`.Data.detailed_balance_factor`<commit_msg>DOC: Updated examples to reflect changes to Data class<commit_after>Data handling with the Data Class
=================================

*Note: This module is still a work-in-progress and the usage of these classes and/or functions may change in the future.*

The following contains examples of how to use the various features in the :py:class:`.Data`, whose purpose is to simplify the handling of neutron scattering data and provide basic analysis and visualization functions.

Initialization: loading data
----------------------------
There are a couple of options to initialize the :py:class:`.Data` object. If your data is in SPICE (HFIR, ORNL), ICE or ICP (NCNR) formats then you can use :py:meth:`.load`, otherwise you will need to load the data and pass it to :py:class:`.Data` manually.

Load from file
^^^^^^^^^^^^^^
First, let's assume that you have a data file, ``'scan.dat'`` from a HFIR instrument (SPICE format):

>>> from neutronpy import load
>>> data = load('scan.dat')

This builds ``data`` automatically and loads the following information from the file:

* detector counts
* monitor counts
* h, k, l
* energy transfer
* temperature
* time
* ei, ef (future feature)

Currently, other data from the files is not included, but some additional support will be added in the future.

.. note::
    Note: The primary problem is there is no standard variable name set between different file formats, and documentation for the file formats are not complete in some cases. Additionally, instruments at the same facility often use different software, so formats are frequently incompatible.

If you want to load more than one file at a time, simple add the file names as separate arguments, *e.g.*

>>> data = load('scan1.dat', 'scan2.dat')

By default, :py:meth:`.load` attempts to determine the format of the input files automatically, but you can specify the ``mode`` if desired. Valid modes are currently:

* ``'SPICE'`` - HFIR files
* ``'ICE'`` - NCNR files
* ``'ICP'`` - NCNR files
* ``'iexy'`` - plaintext files from *e.g.* DAVE (future support planned)

Pass pre-loaded data
^^^^^^^^^^^^^^^^^^^^
Assuming that your data is in a format that is not supported by :py:meth:`.Data.load_file` you will need to load the data yourself and pass it to the :py:class:`.Data` class and build ``Data.Q`` using :py:meth:`.Data.build_Q`. To build ``Data.Q`` you must have defined ``h``, ``k``, ``l``, ``e``, and ``temp``.

>>> data = Data(h=h, k=k, l=l, e=e, temp=temp, detector=detector, monitor=monitor, time=time)

``Data`` properties
---------------
Below I outline some of the most common properties that you will want of the :py:class:`.Data` class.

Intensity and error
^^^^^^^^^^^^^^^^^^^
Intensity, *i.e.* ``detector / monitor * m0`` and square-root error are respectively obtained by

>>> data.intensity
>>> data.error

Monitor normalization
^^^^^^^^^^^^^^^^^^^^^
If you want to normalize to a particular monitor ``m0`` then you will need to define it, *e.g.*

>>> data.m0 = 1e5

If you do not choose a ``m0``, when you call :py:meth:`.Data.intensity` one will be defined for you based on the ``monitor`` already defined in ``data``.

Time normalization
^^^^^^^^^^^^^^^^^^
If you want to normalize to a particular time ``t0`` then you will need to set ``time_norm`` to ``True`` and define ``t0`` in minutes, *e.g.*

>>> data.time_norm = True
>>> data.t0 = 5

If you do not choose a ``t0``, when you call :py:meth:`.Data.intensity` one will be defined for you based on the ``time`` already defined in ``data``.

The ``Q`` vector
^^^^^^^^^^^^^^^^
In this case, ``Q`` is collection of column arrays defined as ``[h, k, l, e, temp]``, with ``data.Q.shape = (N, 5)``. Typically, one would expect that ``temp`` not be included in ``Q``, but for the purposes of rebinning it is included currently. *In the future, rebinning may be expanded to include other arbitrary dimensions, rather than just these five.* If data has been loaded from one of the supported file formats, or :py:meth:`.Data.build_Q` has been used then these variables can also be accessed separately by:

>>> h = data.h
>>> k = data.k
>>> l = data.l
>>> e = data.e
>>> temp = data.temp

``Data`` operations
-------------------
Combining data is as easy as adding multiple ``Data`` objects together, *e.g.*

>>> data1 = load('scan1.dat', mode='SPICE')
>>> data2 = load('scan2.dat', mode='SPICE')
>>> data = data1 + data2

This will combine monitor and detector counts for existing points and concatenate unique points in the two objects to create a new ``data`` object.

Subtracting works in a similar way, but keep in mind that in its current form it doesn't interpolate, so if ``Q`` is different between the two ``data`` variables then you will end up with negative intensities at positions where there isn't an overlapping Q. Proper background subtraction will be implemented in the future.

The ``*``, ``/`` and ``**`` operators only act on the detector variable. This is useful for example if you want to apply the detailed balance factor obtained from :py:meth:`.Data.detailed_balance_factor`
{'fraction_non_alphanumeric': 0.14677267496600063, 'fraction_numerical': 0.0038706977717334447, 'mean_word_length': 4.328874024526199, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 111, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4799837', 'n_tokens_mistral': 2676, 'n_tokens_neox': 2591, 'n_words': 1329}