<commit_before>[[delivery-metrics]]
=== Delivery metrics

One of the core features of Ontrack is the assignment of <<model,promotion levels>> to some builds, either explicitly from the CI or through <<promotion-levels-auto-promotion,auto promotion>>.

_Delivery metrics_ are about measuring the performance and stability of these promotions on four different axes:

* _lead time to promotion_ - how long does it take from the moment a <<model,build>> is created to the moment it is promoted to a given level? This gives an indication on the performance of your delivery.

* _frequency_ - how many promotions do you get over a given period of time? This gives an absolute indicator about how often your delivery process performs.

* _success rate_ - what is the percentage of builds reaching a given promotion level? While 100% is not a goal (because of the very nature of a delivery pipeline, where failure is expected when finding actual issues), high values indicate a high stability of your delivery process.

* _time to restore_ - given a build which is _not_ promoted, how long does it take to restore this promotion? The time it takes to _fix_ an issue is a strong indicator of the resilience of your delivery process.

These metrics are valid for:

* one project
* one branch
* one promotion level

Additionally, Ontrack distinguishes between:

* single project metrics - where we measure the performance of a promotion level within the same project
* end-to-end project metrics - where we measure the performance of a promotion level _across_ several projects, by following the <<builds-links,build links>>. Single project metrics are a special case of end-to-end project metrics.

[[delivery-metrics-single-project]]
==== Single project delivery metrics

Navigate to any promotion level. Four charts are displayed, one for each of the axes:

* lead time, with mean, 90th percentile & maximum
* frequency, count of promotions over the selected period
* success rate
* time to restore, with mean, 90th percentile & maximum

image::images/delivery-metrics-promotion-charts.png[Promotion charts]

For each chart, you can:

* visualize the data
* export the chart as an image
* put the chart fullscreen

You can select the interval and the period (cog icon next to the chart title). These settings are valid for all the charts for all the promotions.

Additionally, you also have two charts a validation stamp level:

image::images/delivery-metrics-validation-charts.png[Validation charts]

* how long lasts the validation?
* how stable it is?

[NOTE]
====
For API access to the single project delivery metrics, consider using the exported <<delivery-metrics-e2e>>.
====

[[delivery-metrics-e2e]]
==== End-to-end project delivery metrics<commit_msg>#981 doc: delivery metrics for end-to-end dependencies<commit_after>[[delivery-metrics]]
=== Delivery metrics

One of the core features of Ontrack is the assignment of <<model,promotion levels>> to some builds, either explicitly from the CI or through <<promotion-levels-auto-promotion,auto promotion>>.

_Delivery metrics_ are about measuring the performance and stability of these promotions on four different axes:

* _lead time to promotion_ - how long does it take from the moment a <<model,build>> is created to the moment it is promoted to a given level? This gives an indication on the performance of your delivery.

* _frequency_ - how many promotions do you get over a given period of time? This gives an absolute indicator about how often your delivery process performs.

* _success rate_ - what is the percentage of builds reaching a given promotion level? While 100% is not a goal (because of the very nature of a delivery pipeline, where failure is expected when finding actual issues), high values indicate a high stability of your delivery process.

* _time to restore_ - given a build which is _not_ promoted, how long does it take to restore this promotion? The time it takes to _fix_ an issue is a strong indicator of the resilience of your delivery process.

These metrics are valid for:

* one project
* one branch
* one promotion level

Additionally, Ontrack distinguishes between:

* single project metrics - where we measure the performance of a promotion level within the same project
* end-to-end project metrics - where we measure the performance of a promotion level _across_ several projects, by following the <<builds-links,build links>>. Single project metrics are a special case of end-to-end project metrics.

[[delivery-metrics-single-project]]
==== Single project delivery metrics

Navigate to any promotion level. Four charts are displayed, one for each of the axes:

* lead time, with mean, 90th percentile & maximum
* frequency, count of promotions over the selected period
* success rate
* time to restore, with mean, 90th percentile & maximum

image::images/delivery-metrics-promotion-charts.png[Promotion charts]

For each chart, you can:

* visualize the data
* export the chart as an image
* put the chart fullscreen

You can select the interval and the period (cog icon next to the chart title). These settings are valid for all the charts for all the promotions.

Additionally, you also have two charts a validation stamp level:

image::images/delivery-metrics-validation-charts.png[Validation charts]

* how long lasts the validation?
* how stable it is?

[NOTE]
====
For API access to the single project delivery metrics, consider using the exported <<delivery-metrics-e2e>>.
====

[[delivery-metrics-e2e]]
==== End-to-end project delivery metrics

[NOTE]
====
As of now, end-to-end delivery metrics are not available in the UI, only as <<operations-metrics,metrics>>. You'll need to use tools like Grafana or Kibana to show them on charts.
====

The end-to-end delivery metrics are performance metrics for the promotions across several projects, following the links between the builds.

For example:

* given a project P depending on a component C which itself depends on a library L
* given a promotion GOLD valid on all these components

We can measure the performance of this promotion by following the links from L to C to P.

The metric axes exposed at the beginning translate into:

* lead time - how long does it take from the moment a L build is created to the moment it is available in C and P, with all builds are all levels being GOLD?
* frequency - how often does it happen that for each L build, all linked L, C and P builds will all be GOLD?
* success rate - for each L build, how many of them are GOLD and also their linked C and P builds?
* time to restore - if a chain L -> C -> P is not GOLD, how long does it take to restore it to full GOLD?

All these metrics are <<operations-metrics,exported>> for each transitive link, branch & promotion. In our previous example, we'll have records for the pairs:

* L:L - same as <<delivery-metrics-single-project,single project delivery metric>>
* L:C, C:P - one level of dependency
* L:P - two levels of dependency

The following metrics are available:

* ontrack_dm_promotion_lead_time - in seconds - for the lead time
ontrack_dm_promotion_success_rate - percentage (from 0.0 to 1.0) - success rate
ontrack_dm_promotion_ttr - in seconds - for the time to restore

Each metric is associated with the following tags:

* sourceProject, sourceBranch
* targetProject, targetBranch
* promotion
{'fraction_non_alphanumeric': 0.05932779970063954, 'fraction_numerical': 0.003401823377330249, 'mean_word_length': 4.530474040632054, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3305369', 'n_tokens_mistral': 1868, 'n_tokens_neox': 1801, 'n_words': 1056}