<commit_before>.. SPDX-License-Identifier: CC-BY-4.0

Overview of the Functest suites
===============================

Functest is the OPNFV project primarily targeting functional testing.
In the Continuous Integration pipeline, it is launched after an OPNFV fresh
installation to validate and verify the basic functions of the
infrastructure.

The current list of test suites can be distributed over 4 main domains:
  * VIM (Virtualised Infrastructure Manager)
  * Controllers (i.e. SDN Controllers)
  * Features
  * VNF (Virtual Network Functions)

Functest test suites are also distributed in the OPNFV testing categories:
healthcheck, smoke, features, components, performance, VNF, Stress tests.

All the Healthcheck and smoke tests of a given scenario must be succesful to
validate the scenario for the release.

+-------------+---------------+------------+----------------------------------+
| Domain      | Tier          | Test case  | Comments                         |
+=============+===============+============+==================================+
| VIM         | healthcheck   | connection | Check OpenStack connectivity     |
|             |               | \_check    | through SNAPS framework          |
|             |               +------------+----------------------------------+
|             |               | api_check  | Check OpenStack API through      |
|             |               |            | SNAPS framework                  |
|             |               +------------+----------------------------------+
|             |               | snaps      | basic instance creation, check   |
|             |               | \_health   | DHCP                             |
|             |               | \_check    |                                  |
|             +---------------+------------+----------------------------------+
|             | smoke         | vping_ssh  | NFV "Hello World" using an SSH   |
|             |               |            | connection to a destination VM   |
|             |               |            | over a created floating IP       |
|             |               |            | address on the SUT Public /      |
|             |               |            | External network. Using the SSH  |
|             |               |            | connection a test script is then |
|             |               |            | copied to the destination        |
|             |               |            | VM and then executed via SSH.    |
|             |               |            | The script will ping another     |
|             |               |            | VM on a specified IP address over|
|             |               |            | the SUT Private Tenant network   |
|             |               +------------+----------------------------------+
|             |               | vping      | Uses Ping with given userdata    |
|             |               | \_userdata | to test intra-VM connectivity    |
|             |               |            | over the SUT Private Tenant      |
|             |               |            | network. The correct operation   |
|             |               |            | of the NOVA Metadata service is  |
|             |               |            | also verified in this test       |
|             |               +------------+----------------------------------+
|             |               | tempest    | Generate and run a relevant      |
|             |               | \_smoke    | Tempest Test Suite in smoke mode.|
|             |               | \_serial   | The generated test set is        |
|             |               |            | dependent on the OpenStack       |
|             |               |            | deployment environment           |
|             |               +------------+----------------------------------+
|             |               | rally      | Run a subset of the OpenStack    |
|             |               | \_sanity   | Rally Test Suite in smoke mode   |
|             |               +------------+----------------------------------+
|             |               | snaps\     | Run the SNAPS-OO integration     |
|             |               | \_smoke    | tests                            |
|             |               +------------+----------------------------------+
|             |               | refstack   | Reference RefStack suite         |
|             |               | \_defcore  | tempest selection for NFV        |
|             |               +------------+----------------------------------+
|             |               | patrole    | Patrole is a tempest plugin for  |
|             |               |            | testing and verifying RBAC policy|
|             |               |            | enforcement, which offers testing|
|             |               |            | for the following OpenStack      |
|             |               |            | services: Nova, Neutron, Glance, |
|             |               |            | Cinder and Keystone              |
|             +---------------+------------+----------------------------------+
|             |               | neutron    | The neutron trunk port testcases |
|             |               | \_trunk    | have been introduced and they are|
|             |               |            | supported by installers :        |
|             |               |            | Apex, Fuel and Compass.          |
|             +---------------+------------+----------------------------------+
|             | components    | tempest    | Generate and run a full set of   |
|             |               | \_full     | the OpenStack Tempest Test Suite.|
|             |               | \_parallel | See the OpenStack reference test |
|             |               |            | suite `[2]`_. The generated      |
|             |               |            | test set is dependent on the     |
|             |               |            | OpenStack deployment environment |
|             |               +------------+----------------------------------+
|             |               | rally_full | Run the OpenStack testing tool   |
|             |               |            | benchmarking OpenStack modules   |
|             |               |            | See the Rally documents `[3]`_   |
+-------------+---------------+------------+----------------------------------+
| Controllers | smoke         | odl        | Opendaylight Test suite          |
|             |               |            | Limited test suite to check the  |
|             |               |            | basic neutron (Layer 2)          |
|             |               |            | operations mainly based on       |
|             |               |            | upstream testcases. See below    |
|             |               |            | for details                      |
|             |               +------------+----------------------------------+
|             |               | odl        | Test Suite for the OpenDaylight  |
|             |               | \_netvirt  | SDN Controller when the NetVirt  |
|             |               |            | features are installed. It       |
|             |               |            | integrates some test suites from |
|             |               |            | upstream using Robot as the test |
|             |               |            | framework                        |
+-------------+---------------+------------+----------------------------------+
| Features    | features      | bgpvpn     | Implementation of the OpenStack  |
|             |               |            | bgpvpn API from the SDNVPN       |
|             |               |            | feature project. It allows for   |
|             |               |            | the creation of BGP VPNs.        |
|             |               |            | See `SDNVPN User Guide`_ for     |
|             |               |            | details                          |
|             |               +------------+----------------------------------+
|             |               | doctor     | Doctor platform, as of Colorado  |
|             |               |            | release, provides the three      |
|             |               |            | features:                        |
|             |               |            | * Immediate Notification         |
|             |               |            | * Consistent resource state      |
|             |               |            | awareness for compute host down  |
|             |               |            | * Valid compute host status      |
|             |               |            | given to VM owner                |
|             |               |            | See `Doctor User Guide`_ for     |
|             |               |            | details                          |
|             |               +------------+----------------------------------+
|             |               | odl-sfc    | SFC testing for odl scenarios    |
|             |               |            | See `SFC User Guide`_ for details|
|             |               +------------+----------------------------------+
|             |               | parser     | Parser is an integration project |
|             |               |            | which aims to provide            |
|             |               |            | placement/deployment templates   |
|             |               |            | translation for OPNFV platform,  |
|             |               |            | including TOSCA -> HOT, POLICY ->|
|             |               |            | TOSCA and YANG -> TOSCA. it      |
|             |               |            | deals with a fake vRNC.          |
|             |               |            | See `Parser User Guide`_ for     |
|             |               |            | details                          |
|             |               +------------+----------------------------------+
|             |               | fds        | Test Suite for the OpenDaylight  |
|             |               |            | SDN Controller when the GBP      |
|             |               |            | features are installed. It       |
|             |               |            | integrates some test suites from |
|             |               |            | upstream using Robot as the test |
|             |               |            | framework                        |
+-------------+---------------+------------+----------------------------------+
| VNF         | vnf           | cloudify   | Example of a real VNF deployment |
|             |               | \_ims      | to show the NFV capabilities of  |
|             |               |            | the platform. The IP Multimedia  |
|             |               |            | Subsytem is a typical Telco test |
|             |               |            | case, referenced by ETSI.        |
|             |               |            | It provides a fully functional   |
|             |               |            | VoIP System                      |
|             |               +------------+----------------------------------+
|             |               | vyos       | vRouter testing                  |
|             |               | \_vrouter  |                                  |
|             |               +------------+----------------------------------+
|             |               | juju_epc   | vEPC validation with Juju as VNF |
|             |               |            | manager and ABoT as test executor|
|             |               +------------+----------------------------------+
|             |               | cloudify   | Based on cloudify_ims test case  |
|             |               | \_ims_perf | cloudify_ims_perf substitutes    |
|             |               |            | the signaling test suite by an   |
|             |               |            | automatic deployment of an Ixia  |
|             |               |            | loader and generic SIP stress    |
|             |               |            | tests.                           |
|             |               |            | This work has been initiated     |
|             |               |            | during the plugfest and allows   |
|             |               |            | realistic load tests on top of   |
|             |               |            | cloudify_ims. Please note that   |
|             |               |            | this test is available but not   |
|             |               |            | declared in testcases.yaml as it |
|             |               |            | requires access to proprietary   |
|             |               |            | resources (Ixia loader)          |
+-------------+---------------+------------+----------------------------------+


As shown in the above table, Functest is structured into different 'domains',
'tiers' and 'test cases'. Each 'test case' usually represents an actual
'Test Suite' comprised -in turn- of several test cases internally.

Test cases also have an implicit execution order. For example, if the early
'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
Tier testcases, there is little point to launch a full testcase execution
round.

In Danube, we merged smoke and sdn controller tiers in smoke tier.

An overview of the Functest Structural Concept is depicted graphically below:

.. figure:: ../../../images/concepts_mapping_final.png
   :align: center
   :alt: Functest Concepts Structure

Some of the test cases are developed by Functest team members, whereas others
are integrated from upstream communities or other OPNFV projects. For example,
`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
OpenStack integration test suite and Functest is in charge of the selection,<commit_msg>Add Kubernetes tests to userguide<commit_after>.. SPDX-License-Identifier: CC-BY-4.0

Overview of the Functest suites
===============================

Functest is the OPNFV project primarily targeting functional testing.
In the Continuous Integration pipeline, it is launched after an OPNFV fresh
installation to validate and verify the basic functions of the
infrastructure.

The current list of test suites can be distributed over 5 main domains:
  * VIM (Virtualised Infrastructure Manager)
  * Controllers (i.e. SDN Controllers)
  * Features
  * VNF (Virtual Network Functions)
  * Kubernetes

Functest test suites are also distributed in the OPNFV testing categories:
healthcheck, smoke, features, components, performance, VNF, Stress tests.

All the Healthcheck and smoke tests of a given scenario must be succesful to
validate the scenario for the release.

+-------------+---------------+------------+----------------------------------+
| Domain      | Tier          | Test case  | Comments                         |
+=============+===============+============+==================================+
| VIM         | healthcheck   | connection | Check OpenStack connectivity     |
|             |               | \_check    | through SNAPS framework          |
|             |               +------------+----------------------------------+
|             |               | api_check  | Check OpenStack API through      |
|             |               |            | SNAPS framework                  |
|             |               +------------+----------------------------------+
|             |               | snaps      | basic instance creation, check   |
|             |               | \_health   | DHCP                             |
|             |               | \_check    |                                  |
|             +---------------+------------+----------------------------------+
|             | smoke         | vping_ssh  | NFV "Hello World" using an SSH   |
|             |               |            | connection to a destination VM   |
|             |               |            | over a created floating IP       |
|             |               |            | address on the SUT Public /      |
|             |               |            | External network. Using the SSH  |
|             |               |            | connection a test script is then |
|             |               |            | copied to the destination        |
|             |               |            | VM and then executed via SSH.    |
|             |               |            | The script will ping another     |
|             |               |            | VM on a specified IP address over|
|             |               |            | the SUT Private Tenant network   |
|             |               +------------+----------------------------------+
|             |               | vping      | Uses Ping with given userdata    |
|             |               | \_userdata | to test intra-VM connectivity    |
|             |               |            | over the SUT Private Tenant      |
|             |               |            | network. The correct operation   |
|             |               |            | of the NOVA Metadata service is  |
|             |               |            | also verified in this test       |
|             |               +------------+----------------------------------+
|             |               | tempest    | Generate and run a relevant      |
|             |               | \_smoke    | Tempest Test Suite in smoke mode.|
|             |               | \_serial   | The generated test set is        |
|             |               |            | dependent on the OpenStack       |
|             |               |            | deployment environment           |
|             |               +------------+----------------------------------+
|             |               | rally      | Run a subset of the OpenStack    |
|             |               | \_sanity   | Rally Test Suite in smoke mode   |
|             |               +------------+----------------------------------+
|             |               | snaps\     | Run the SNAPS-OO integration     |
|             |               | \_smoke    | tests                            |
|             |               +------------+----------------------------------+
|             |               | refstack   | Reference RefStack suite         |
|             |               | \_defcore  | tempest selection for NFV        |
|             |               +------------+----------------------------------+
|             |               | patrole    | Patrole is a tempest plugin for  |
|             |               |            | testing and verifying RBAC policy|
|             |               |            | enforcement, which offers testing|
|             |               |            | for the following OpenStack      |
|             |               |            | services: Nova, Neutron, Glance, |
|             |               |            | Cinder and Keystone              |
|             +---------------+------------+----------------------------------+
|             |               | neutron    | The neutron trunk port testcases |
|             |               | \_trunk    | have been introduced and they are|
|             |               |            | supported by installers :        |
|             |               |            | Apex, Fuel and Compass.          |
|             +---------------+------------+----------------------------------+
|             | components    | tempest    | Generate and run a full set of   |
|             |               | \_full     | the OpenStack Tempest Test Suite.|
|             |               | \_parallel | See the OpenStack reference test |
|             |               |            | suite `[2]`_. The generated      |
|             |               |            | test set is dependent on the     |
|             |               |            | OpenStack deployment environment |
|             |               +------------+----------------------------------+
|             |               | rally_full | Run the OpenStack testing tool   |
|             |               |            | benchmarking OpenStack modules   |
|             |               |            | See the Rally documents `[3]`_   |
+-------------+---------------+------------+----------------------------------+
| Controllers | smoke         | odl        | Opendaylight Test suite          |
|             |               |            | Limited test suite to check the  |
|             |               |            | basic neutron (Layer 2)          |
|             |               |            | operations mainly based on       |
|             |               |            | upstream testcases. See below    |
|             |               |            | for details                      |
|             |               +------------+----------------------------------+
|             |               | odl        | Test Suite for the OpenDaylight  |
|             |               | \_netvirt  | SDN Controller when the NetVirt  |
|             |               |            | features are installed. It       |
|             |               |            | integrates some test suites from |
|             |               |            | upstream using Robot as the test |
|             |               |            | framework                        |
+-------------+---------------+------------+----------------------------------+
| Features    | features      | bgpvpn     | Implementation of the OpenStack  |
|             |               |            | bgpvpn API from the SDNVPN       |
|             |               |            | feature project. It allows for   |
|             |               |            | the creation of BGP VPNs.        |
|             |               |            | See `SDNVPN User Guide`_ for     |
|             |               |            | details                          |
|             |               +------------+----------------------------------+
|             |               | doctor     | Doctor platform, as of Colorado  |
|             |               |            | release, provides the three      |
|             |               |            | features:                        |
|             |               |            | * Immediate Notification         |
|             |               |            | * Consistent resource state      |
|             |               |            | awareness for compute host down  |
|             |               |            | * Valid compute host status      |
|             |               |            | given to VM owner                |
|             |               |            | See `Doctor User Guide`_ for     |
|             |               |            | details                          |
|             |               +------------+----------------------------------+
|             |               | odl-sfc    | SFC testing for odl scenarios    |
|             |               |            | See `SFC User Guide`_ for details|
|             |               +------------+----------------------------------+
|             |               | parser     | Parser is an integration project |
|             |               |            | which aims to provide            |
|             |               |            | placement/deployment templates   |
|             |               |            | translation for OPNFV platform,  |
|             |               |            | including TOSCA -> HOT, POLICY ->|
|             |               |            | TOSCA and YANG -> TOSCA. it      |
|             |               |            | deals with a fake vRNC.          |
|             |               |            | See `Parser User Guide`_ for     |
|             |               |            | details                          |
|             |               +------------+----------------------------------+
|             |               | fds        | Test Suite for the OpenDaylight  |
|             |               |            | SDN Controller when the GBP      |
|             |               |            | features are installed. It       |
|             |               |            | integrates some test suites from |
|             |               |            | upstream using Robot as the test |
|             |               |            | framework                        |
+-------------+---------------+------------+----------------------------------+
| VNF         | vnf           | cloudify   | Example of a real VNF deployment |
|             |               | \_ims      | to show the NFV capabilities of  |
|             |               |            | the platform. The IP Multimedia  |
|             |               |            | Subsytem is a typical Telco test |
|             |               |            | case, referenced by ETSI.        |
|             |               |            | It provides a fully functional   |
|             |               |            | VoIP System                      |
|             |               +------------+----------------------------------+
|             |               | vyos       | vRouter testing                  |
|             |               | \_vrouter  |                                  |
|             |               +------------+----------------------------------+
|             |               | juju_epc   | vEPC validation with Juju as VNF |
|             |               |            | manager and ABoT as test executor|
|             |               +------------+----------------------------------+
|             |               | cloudify   | Based on cloudify_ims test case  |
|             |               | \_ims_perf | cloudify_ims_perf substitutes    |
|             |               |            | the signaling test suite by an   |
|             |               |            | automatic deployment of an Ixia  |
|             |               |            | loader and generic SIP stress    |
|             |               |            | tests.                           |
|             |               |            | This work has been initiated     |
|             |               |            | during the plugfest and allows   |
|             |               |            | realistic load tests on top of   |
|             |               |            | cloudify_ims. Please note that   |
|             |               |            | this test is available but not   |
|             |               |            | declared in testcases.yaml as it |
|             |               |            | requires access to proprietary   |
|             |               |            | resources (Ixia loader)          |
+-------------+---------------+------------+----------------------------------+
| Kubernetes  | healthcheck   | k8s_smoke  | Test a running Kubernetes        |
|             |               |            | cluster and ensure it satisfies  |
|             |               |            | minimal functional requirements  |
|             +---------------+------------+----------------------------------+
|             | smoke         | k8s\_      | Run a subset of Kubernetes       |
|             |               | conformance| End-to-End tests, expected to    |
|             |               |            | pass on any Kubernetes cluster   |
|             +---------------+------------+----------------------------------+
|             | stor4nfv      | stor4nfv   | Run tests necessary to           |
|             |               | \_k8s      | demonstrate conformance of the   |
|             |               |            | K8s+Stor4NFV deployment          |
|             +---------------+------------+----------------------------------+
|             | clover        | clover_k8s | Test functionality of            |
|             |               |            | K8s+Istio+Clover deployment.     |
+-------------+---------------+------------+----------------------------------+


As shown in the above table, Functest is structured into different 'domains',
'tiers' and 'test cases'. Each 'test case' usually represents an actual
'Test Suite' comprised -in turn- of several test cases internally.

Test cases also have an implicit execution order. For example, if the early
'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
Tier testcases, there is little point to launch a full testcase execution
round.

In Danube, we merged smoke and sdn controller tiers in smoke tier.

An overview of the Functest Structural Concept is depicted graphically below:

.. figure:: ../../../images/concepts_mapping_final.png
   :align: center
   :alt: Functest Concepts Structure

Some of the test cases are developed by Functest team members, whereas others
are integrated from upstream communities or other OPNFV projects. For example,
`Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
OpenStack integration test suite and Functest is in charge of the selection,
{'fraction_non_alphanumeric': 0.17535300786752328, 'fraction_numerical': 0.0007214759336242141, 'mean_word_length': 0.6535817758336647, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 360, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6123028', 'n_tokens_mistral': 5978, 'n_tokens_neox': 5652, 'n_words': 1718}