<commit_before>Data Files
==========

Data files can be encoded as either `YAML <http://yaml.org>`_ or
`JSON <http://www.json.org>`_: the software deals with both the same way.
We define the data file in two parts which describe:

a) the *independent variables* (e.g. the x-axis of a plot);
b) the *dependent variables* (the thing you're measuring, e.g. the y-axis of a plot).

Each table can have any number of independent and dependent variables
(columns), but each must have the same number of data points (rows).
Independent variables consist of a list of values, each of which generally comprises
``low`` and ``high`` bin limits, together with a central ``value``.  However, the
central ``value`` can be omitted if it coincides with the bin midpoint, while the
``low`` and ``high`` bin limits can be omitted if they are not applicable.
If there are no independent variables, for example, an inclusive cross-section
measurement, an empty list should be specified, ``independent_variables: []``.

Each variable comprises a header (the column name) and a list of values
(the rows in your table).  The header should define the variable
including units unless the variable is dimensionless.
For the dependent variables, you can also
define *qualifiers*.  These are extra metadata describing the
measurement, such as the energy, the reaction type, and possible
kinematic cuts on variables such as transverse momentum and
(pseudo)rapidity.


YAML data file example
----------------------

.. code-block:: yaml

   independent_variables:
   - header: {name: Leading dilepton PT, units: GEV}
     values:
     - {low: 0, high: 60}
     - {low: 60, high: 100}
     - {low: 100, high: 200}
     - {low: 200, high: 600}
   dependent_variables:
   - header: {name: 10**6 * 1/SIG(fiducial) * D(SIG(fiducial))/DPT, units: GEV**-1}
     qualifiers:
     - {name: RE, value: P P --> Z0 < LEPTON+ LEPTON- > Z0 < LEPTON+ LEPTON- > X}
     - {name: SQRT(S), units: GEV, value: 7000}
     values:
     - value: 7000
       errors:
       - {symerror: 1100, label: stat}
       - {symerror: 79, label: 'sys,detector'}
       - {symerror: 15, label: 'sys,background'}
     - value: 9800
       errors:
       - {symerror: 1600, label: stat}
       - {symerror: 75, label: 'sys,detector'}
       - {symerror: 15, label: 'sys,background'}
     - value: 1600
       errors:
       - {symerror: 490, label: stat}
       - {symerror: 41, label: 'sys,detector'}
       - {symerror: 2, label: 'sys,background'}
     - value: 80
       errors:
       - {symerror: 60, label: stat}
       - {symerror: 2, label: 'sys,detector'}
       - {symerror: 0, label: 'sys,background'}


Uncertainties
-------------

Multiple uncertainties can be assigned to each data point, each with
an optional label to distinguish them.  There are two main classes
of uncertainty that can be encoded: symmetric errors and
asymmetric errors.  A symmetric error allows you to specify plus
and minus errors using one value, e.g. ``symerror: 0.4``, while an
asymmetric error allows both plus and minus errors to be explicitly
encoded, e.g. ``asymerror: {plus: 0.4, minus: -0.3}``.  Note that here
"plus" and "minus" can refer to "up" and "down" variations of the
source of uncertainty, and do not necessarily match the sign of the
resultant uncertainty on the measurement (which can change sign along a
distribution).  Note that ``symerror: 0.4`` is equivalent to
``asymerror: {plus: 0.4, minus: -0.4}``.  The opposite-sign case,
``symerror: -0.4`` is equivalent to ``asymerror: {plus: -0.4, minus: 0.4}``.
A one-sided uncertainty can be represented
using an empty string, e.g. ``asymerror: {plus: '', minus: -0.3}``.
Error values are normally taken as absolute, but relative errors
can be specified by including a ``%`` symbol after the number to define
the error as a percentage of the central ``value``.

Within the context of the
`LHC Electroweak Working Group <https://twiki.cern.ch/twiki/bin/view/LHCPhysics/LHCEW>`_,
it has been proposed (see
`talk <https://indico.cern.ch/event/779259/contributions/3242594/attachments/1770317/2876299/LCorpe_LHCEWWG_Correlations_131218.pdf>`_)
to provide a breakdown of individual uncertainty contributions rather
than a correlation/covariance matrix for systematic uncertainties.
However, a statistical correlation matrix will still be needed.<commit_msg>docs: add paragraph on check on zero uncertainties<commit_after>Data Files
==========

Data files can be encoded as either `YAML <http://yaml.org>`_ or
`JSON <http://www.json.org>`_: the software deals with both the same way.
We define the data file in two parts which describe:

a) the *independent variables* (e.g. the x-axis of a plot);
b) the *dependent variables* (the thing you're measuring, e.g. the y-axis of a plot).

Each table can have any number of independent and dependent variables
(columns), but each must have the same number of data points (rows).
Independent variables consist of a list of values, each of which generally comprises
``low`` and ``high`` bin limits, together with a central ``value``.  However, the
central ``value`` can be omitted if it coincides with the bin midpoint, while the
``low`` and ``high`` bin limits can be omitted if they are not applicable.
If there are no independent variables, for example, an inclusive cross-section
measurement, an empty list should be specified, ``independent_variables: []``.

Each variable comprises a header (the column name) and a list of values
(the rows in your table).  The header should define the variable
including units unless the variable is dimensionless.
For the dependent variables, you can also
define *qualifiers*.  These are extra metadata describing the
measurement, such as the energy, the reaction type, and possible
kinematic cuts on variables such as transverse momentum and
(pseudo)rapidity.


YAML data file example
----------------------

.. code-block:: yaml

   independent_variables:
   - header: {name: Leading dilepton PT, units: GEV}
     values:
     - {low: 0, high: 60}
     - {low: 60, high: 100}
     - {low: 100, high: 200}
     - {low: 200, high: 600}
   dependent_variables:
   - header: {name: 10**6 * 1/SIG(fiducial) * D(SIG(fiducial))/DPT, units: GEV**-1}
     qualifiers:
     - {name: RE, value: P P --> Z0 < LEPTON+ LEPTON- > Z0 < LEPTON+ LEPTON- > X}
     - {name: SQRT(S), units: GEV, value: 7000}
     values:
     - value: 7000
       errors:
       - {symerror: 1100, label: stat}
       - {symerror: 79, label: 'sys,detector'}
       - {symerror: 15, label: 'sys,background'}
     - value: 9800
       errors:
       - {symerror: 1600, label: stat}
       - {symerror: 75, label: 'sys,detector'}
       - {symerror: 15, label: 'sys,background'}
     - value: 1600
       errors:
       - {symerror: 490, label: stat}
       - {symerror: 41, label: 'sys,detector'}
       - {symerror: 2, label: 'sys,background'}
     - value: 80
       errors:
       - {symerror: 60, label: stat}
       - {symerror: 2, label: 'sys,detector'}
       - {symerror: 0, label: 'sys,background'}


Uncertainties
-------------

Multiple uncertainties can be assigned to each data point, each with
an optional label to distinguish them.  There are two main classes
of uncertainty that can be encoded: symmetric errors and
asymmetric errors.  A symmetric error allows you to specify plus
and minus errors using one value, e.g. ``symerror: 0.4``, while an
asymmetric error allows both plus and minus errors to be explicitly
encoded, e.g. ``asymerror: {plus: 0.4, minus: -0.3}``.  Note that here
"plus" and "minus" can refer to "up" and "down" variations of the
source of uncertainty, and do not necessarily match the sign of the
resultant uncertainty on the measurement (which can change sign along a
distribution).  Note that ``symerror: 0.4`` is equivalent to
``asymerror: {plus: 0.4, minus: -0.4}``.  The opposite-sign case,
``symerror: -0.4`` is equivalent to ``asymerror: {plus: -0.4, minus: 0.4}``.
A one-sided uncertainty can be represented
using an empty string, e.g. ``asymerror: {plus: '', minus: -0.3}``.
Error values are normally taken as absolute, but relative errors
can be specified by including a ``%`` symbol after the number to define
the error as a percentage of the central ``value``.

The `hepdata-validator <https://github.com/HEPData/hepdata-validator>`_ (v0.2.0 or greater)
code will invalidate bins where all uncertainties are zero.  This check was introduced to
avoid `problems <https://gitlab.com/hepcedar/rivet/-/issues/69>`_ in fitting applications.
Bins with zero content should preferably be omitted completely from the HEPData table.
Alternatively, missing bins can be indicated with a non-numerical central value like ``'-'``
or an empty string ``''`` and no uncertainties.  In this case, the ``errors`` key should
either be omitted completely or specified as an empty list ``errors: []``.

Within the context of the
`LHC Electroweak Working Group <https://twiki.cern.ch/twiki/bin/view/LHCPhysics/LHCEW>`_,
it has been proposed (see
`talk <https://indico.cern.ch/event/779259/contributions/3242594/attachments/1770317/2876299/LCorpe_LHCEWWG_Correlations_131218.pdf>`_)
to provide a breakdown of individual uncertainty contributions rather
than a correlation/covariance matrix for systematic uncertainties.
However, a statistical correlation matrix will still be needed.
{'fraction_non_alphanumeric': 0.11503477795612627, 'fraction_numerical': 0.027073301230604602, 'mean_word_length': 3.624443344878773, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1411646', 'n_tokens_mistral': 3003, 'n_tokens_neox': 2727, 'n_words': 1198}