<commit_before>.. index:: results introduction

.. _results_intro:

Introduction to Results in LAVA
###############################

.. seealso:: :ref:`test_definition_yaml`

Results in LAVA
***************

At the lowest level, a result in LAVA is called a Test Case and is a simple
flag:

* pass
* fail
* skip
* unknown

In addition, each test case can include a measurement (integer or floating
point), with units.

Test cases are aggregated into a Test Suite. Multiple test suites can be
generated for each submitted test job.

.. seealso:: :ref:`recording_test_result_data`

.. _results_rest_api:

Accessing results
*****************

LAVA exposes a REST API to provide access to the results from test jobs. The
URL is built from the following components:

#. The instance name
#. The ``results`` directory.
#. The test job ID

From this point, all the results for the test job can be downloaded as
:abbr:`CSV (Comma Separated Values)` or YAML.

There are also CSV and YAML download links for the complete set of results for
the job. The export includes details of the test definition names. For example,
if a test job on ``validation.linaro.org`` has the job ID ``1109234``, the CSV
download link for all results would be
``https://validation.linaro.org/results/1109234/csv``. The YAML download link
for all results would be
``https://validation.linaro.org/results/1109234/yaml``.

Accessing specific test results
===============================

Within the results for the entire test job, results are split into test suites,
optionally into test sets and finally into test cases.

Test Suite
----------

The name of the Test Suite is determined by the test job definition. The Test
Suite is created when results are generated in the running test job.

.. include:: examples/test-jobs/qemu-pipeline-first-job.yaml
   :code: yaml
   :start-after: prompts: ["root@debian:"]

In this test job definition, there are two entries in the ``definitions`` list,
so two test suites. The first test suite in the list has the prefix ``0_`` and
the prefix increments for subsequent test suites in the list. The ``name``
element is then appended to create the test suite name.

* ``0_smoke-tests``
* ``1_singlenode-advanced``

Results are only generated when the Lava Test Shell Definition makes at least
one call to ``lava-test-case``.

The test suite can then be appended to the REST API URL for the results to
limit the results to just that test suite:
``/results/1109234/0_smoke-tests/csv`` or:
``/results/1109234/0_smoke-tests/yaml``.

In addition, every test job has a ``lava`` test suite which holds results for
the processing of the pipeline itself. This set of results can hold useful
information like the commit hash of the test definition when it was cloned to
create the overlay, the duration of all actions (including the kernel boot) and
other information.

.. note:: Test definitions will be rejected if the test suite name is set to
   ``lava`` to prevent conflicts with the internal results.

.. _test_set_results:

Test Set
--------

Test Set is optional but allows test writers to subdivide individual results
within a single Lava Test Shell Definition using an arbitrary label.

Some test definitions run the same test with different parameters. To
distinguish between these similar tests, it can be useful to use a test set::

 lava-test-set start syscalls
 lava-test-case syscalls
 # ....
 lava-test-set stop syscalls
 # change parameters
 lava-test-case start math
 lava-test-case math
 # ....
 lava-test-set stop math

This adds a set around those test cases by adding the test set to the URL.
``/results/JOB_ID/2_smoke-tests/syscall_one_test`` becomes
``/results/JOB_ID/2_smoke-tests/syscalls/syscall_one_test``

Test Case
---------

A Test Case can be generated in a number of ways:

* by calling ``lava-test-case`` from a Lava Test Shell.
* during operation of an Action within the pipeline
* by parsing patterns in the test

Each test case has a name and a result. Optionally, test cases can have
measurements and units. The name of the test case **must** be valid as part of
the REST API so whitespace is not allowed.

Accessing the test job logs from results
****************************************

There is a chevron in the test case detail page, directly after the test case<commit_msg>Explain the handling of measurements and Decimal<commit_after>.. index:: results introduction

.. _results_intro:

Introduction to Results in LAVA
###############################

.. seealso:: :ref:`test_definition_yaml`

Results in LAVA
***************

At the lowest level, a result in LAVA is called a Test Case and is a simple
flag:

* pass
* fail
* skip
* unknown

In addition, each test case can include a measurement (integer or floating
point), with units.

Test cases are aggregated into a Test Suite. Multiple test suites can be
generated for each submitted test job.

.. seealso:: :ref:`recording_test_result_data`

.. _results_rest_api:

Accessing results
*****************

LAVA exposes a REST API to provide access to the results from test jobs. The
URL is built from the following components:

#. The instance name
#. The ``results`` directory.
#. The test job ID

From this point, all the results for the test job can be downloaded as
:abbr:`CSV (Comma Separated Values)` or YAML.

There are also CSV and YAML download links for the complete set of results for
the job. The export includes details of the test definition names. For example,
if a test job on ``validation.linaro.org`` has the job ID ``1109234``, the CSV
download link for all results would be
``https://validation.linaro.org/results/1109234/csv``. The YAML download link
for all results would be
``https://validation.linaro.org/results/1109234/yaml``.

Accessing specific test results
===============================

Within the results for the entire test job, results are split into test suites,
optionally into test sets and finally into test cases.

Test Suite
----------

The name of the Test Suite is determined by the test job definition. The Test
Suite is created when results are generated in the running test job.

.. include:: examples/test-jobs/qemu-pipeline-first-job.yaml
   :code: yaml
   :start-after: prompts: ["root@debian:"]

In this test job definition, there are two entries in the ``definitions`` list,
so two test suites. The first test suite in the list has the prefix ``0_`` and
the prefix increments for subsequent test suites in the list. The ``name``
element is then appended to create the test suite name.

* ``0_smoke-tests``
* ``1_singlenode-advanced``

Results are only generated when the Lava Test Shell Definition makes at least
one call to ``lava-test-case``.

The test suite can then be appended to the REST API URL for the results to
limit the results to just that test suite:
``/results/1109234/0_smoke-tests/csv`` or:
``/results/1109234/0_smoke-tests/yaml``.

In addition, every test job has a ``lava`` test suite which holds results for
the processing of the pipeline itself. This set of results can hold useful
information like the commit hash of the test definition when it was cloned to
create the overlay, the duration of all actions (including the kernel boot) and
other information.

.. note:: Test definitions will be rejected if the test suite name is set to
   ``lava`` to prevent conflicts with the internal results.

.. _test_set_results:

Test Set
--------

Test Set is optional but allows test writers to subdivide individual results
within a single Lava Test Shell Definition using an arbitrary label.

Some test definitions run the same test with different parameters. To
distinguish between these similar tests, it can be useful to use a test set::

 lava-test-set start syscalls
 lava-test-case syscalls
 # ....
 lava-test-set stop syscalls
 # change parameters
 lava-test-case start math
 lava-test-case math
 # ....
 lava-test-set stop math

This adds a set around those test cases by adding the test set to the URL.
``/results/JOB_ID/2_smoke-tests/syscall_one_test`` becomes
``/results/JOB_ID/2_smoke-tests/syscalls/syscall_one_test``

Test Case
---------

A Test Case can be generated in a number of ways:

* by calling ``lava-test-case`` from a Lava Test Shell.
* during operation of an Action within the pipeline
* by parsing patterns in the test

Each test case has a name and a result. Optionally, test cases can have
measurements and units. The name of the test case **must** be valid as part of
the REST API so whitespace is not allowed.

If a test case has a measurement, the measurement is represented as a python
``Decimal`` value as this provides the best way to preserve precision through
the various export handlers.

.. code-block:: python

    >>> case.measurement
    Decimal('112.2510000000')
    >>> import decimal
    >>> decimal.Decimal(case.measurement)
    Decimal('112.2510000000')
    >>> decimal.Decimal(case.measurement).normalize()
    Decimal('112.251')
    >>> float(decimal.Decimal(case.measurement).normalize())
    112.251

Representation of some numbers in Decimal can look a little odd but these
can be easily converted or compared:

.. code-block:: python

    >>> decimal.Decimal(0E-10) == 0
    True

.. seealso:: The python documentation on Decimal at
   https://docs.python.org/2.7/library/decimal.html#module-decimal

Accessing the test job logs from results
****************************************

There is a chevron in the test case detail page, directly after the test case
{'fraction_non_alphanumeric': 0.10260741053520532, 'fraction_numerical': 0.01351208698405996, 'mean_word_length': 4.101777059773829, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 21, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4405918', 'n_tokens_mistral': 2699, 'n_tokens_neox': 2510, 'n_words': 1293}