<commit_before>---
kep-number: 0034
title: Leveraging Distributed Tracing to Understand Kubernetes Object Lifecycles
authors:
  - "@Monkeyanator"
editors:
  - "@dashpole"
owning-sig: sig-instrumentation
reviewers:
  - "@Random-Liu"
  - "@bogdandrutu"
approvers:
  - "@brancz"
creation-date: 2018-12-04
last-updated: 2018-12-07
status: provisional
---

# Leveraging Distributed Tracing to Understand Kubernetes Object Lifecycles

## Table of Contents

- [Leveraging Distributed Tracing to Understand Kubernetes Object Lifecycles](#leveraging-distributed-tracing-to-understand-kubernetes-object-lifecycles)
    - [Table of Contents](#table-of-contents)
    - [Summary](#summary)
    - [Motivation](#motivation)
        - [Goals](#goals)
        - [Non-Goals](#non-goals)
    - [Proposal](#proposal)
        - [Architecture](#architecture)
            - [Trace lifecycle](#trace-lifecycle)
            - [Root spans](#root-spans)          
            - [Context propagation](#context-propagation)          
        - [Out-of-tree changes](#out-of-tree-changes)
            - [Mutating admission webhook](#mutating-admission-webhook)
            - [Tracing best-practices documentation](#tracing-best-practices-documentation)
        - [In-tree changes](#in-tree-changes)
            - [Trace utility package](#trace-utility-package)
            - [Basic Kubernetes component trace instrumentation](#basic-kubernetes-component-trace-instrumentation)
    - [Graduation Requirements](#graduation-requirements)
    - [Implementation History](#implementation-history)

## Summary

This Kubernetes Enchancement Proposal (KEP) introduces a model for adding distributed tracing to Kubernetes object lifecycles. The inclusion of this trace instrumentation will mark a significant step in making Kubernetes processes more observable, understandable, and debuggable.


## Motivation

Debugging latency issues in Kubernetes is an involved process. There are existing tools which can be used to isolate these 
issues in Kubernetes, but these methods fall short for various reasons. For instance:

* **Logs** are fragmented, and finding out which process was the bottleneck involves digging through troves of unstructured text. In addition, logs do not offer higher-level insight into overall system behavior without an extensive background on the process of interest. 
* **Events** in Kubernetes are only kept for an hour by default, and don't integrate with visualization of analysis tools. To gain trace-like insights would require a large investment in custom tooling
* **Latency metrics** are gathered in some places, but these don't provide understanding into _why_ a given process was slow

Distributed tracing, on the other hand, provides a single window into latency information from across many components and plugins. Trace data is structured, and there are numerous established backends for visualizing and querying over it. This KEP would make it possible to, for instance, retrieve and visualize all pod startups that took more than 30 seconds, involved an `nginx` container, and which mounted more than two volumes.

In addition, due to the self-healing nature of Kubernetes, regressions wherein latencies are affected but the overall task is eventually accomplished are not uncommon. With our current monitoring architecture, these "soft regressions" are often difficult to observe and diagnose. Collecting structured trace data on per-object latencies would enable us to detect these long-term regressions automatically, and quickly determine their root causes.


### Goals

* Make it possible to visualize the progress of objects across distinct Kubernetes components
* Streamline the tedious processes involved in debugging latency issues in Kubernetes
* Make it possible to identify high-level latency regressions, and attribute them back to offending processes


### Non-Goals

* To replace existing logging, metrics, or the events API
* To trace operations from all Kubernetes resource types in a generic manner (i.e. without manual instrumentation)


## Proposal

### Architecture

#### Trace lifecycle

Kubernetes is unique in that it is constantly reconciling its actual state towards some desired state. As a result, it has no definitive concept of an "operation", which breaks the traditional model for distributed tracing. This raises the question of when to begin traces, and when to end them.

In this proposal, we choose to _only_ trace phases of an object's lifecycle wherein it's correcting from an undesired state to its desired state, and to end the trace when it enters this desired state. This means that the same object will export traces for each reconciliation it undergoes. This decision was made because:

1) These reconciliation phases are where Kubernetes performs actual work that can be traced
2) Prevents traces from spanning the entire life of an object, which can be indefninite
3) Provides a consistent, concrete duration for the root span which remains applicable across object types
4) The time required for Kubernetes controllers to reconcile desired and actual state is, by definition, a measurement of either latency experienced by end-users in the fulfillment of their requests, or down-time due to a disruption

#### Root spans

In the standard model for distributed tracing, there exists a span in each trace that all other spans are descendents of and which extends the length of the entire trace, called the `root span`. 

The Kubernetes component that kicks off an operation might not be the same component that ends it. In this proposal, when we are at the point where we want to end a root span, we craft a span to export which acts as the root span for the trace. For example, when the kubelet updates a pod from `Pending` to `Running`, it creates a root span using the start time of the pod as the start, and the current time as the end.


#### Context propagation

To correlate work done between components as belonging to the same trace, we must pass span context across process boundaries. In traditional distributed systems, this context can be passed down through RPC metadata or HTTP headers. Kubernetes, however, due to its watch-based nature, requires us to attach trace context directly to the target object. 

In this proposal, we choose to propagate this span context as an encoded string an object annotation called `trace.kubernetes.io/context`. This annotation value is regenerated and replaced when an object's trace ends, to achieve the desired behavior from [section one](#trace-lifecycle). 

This proposal chooses to use annotations as a less invasive alternative to adding a field to object metadata, but as this proposal matures, adding trace context to the official API should be considered. 


### In-tree changes

#### Vendoring the OpenCensus trace framework

This KEP proposes the use of the [OpenCensus tracing framework](https://opencensus.io/) to create and export spans to configured backends. The OpenCensus framework was chosen for various reasons: 

1) Provides concrete, tested implementations for creating and exporting spans to diverse backends, rather than providing an API specification, as is the case with [OpenTracing](https://opentracing.io/specification/)
2) [Provides an agent](https://github.com/census-instrumentation/opencensus-service) which enables lazy configuration for exporters, batching of spans, and other features 

#### Adding trace utility package

This package will be able to create spans from the span context embedded in the `trace.kubernetes.io/context` object annotation, in addition to embedding context from spans back into the annotation. This package will facilitate tracing across Kubernetes watches. It will also provide an implementation for exporting the root span for a given reconciliation trace.

This package must be divorced from specific exporters, and allow users to configure the desired tracing backend elsewhere (i.e. on the OC agent).

#### Instrumenting various Kubernetes components with pod startup traces

As we move forward with this KEP, we will use the aforementioned trace utility package to trace pod-related operations across the scheduler and kubelet. In code, this corresponds to creating a span (i.e. `ctx, span := trace.StartSpan(ctx, "Component.SampleSpan")`) at the beginning of an operation, and ending the span afterwards (`span.End()`). All calls to tracing functions will be gated with the `ObjectLifecycleTracing` alpha feature-gate, and will be disabled by default.

OpenCensus ships with plugins to transport trace context across gRPC and HTTP boundaries, which enables us to extend our tracing across the CRI and other internal boundaries.

In OpenCensus' Go implementation, span context is passed down through Go context. This will necessitate the threading of context across more of the Kubernetes codebase, which is a [desired outcome regardless](https://github.com/kubernetes/kubernetes/issues/815).

Following these intial pod-related trace additions, trace instrumentation should be added in an ad-hoc fashion to the various Kubernetes components.


### Out-of-tree changes

#### Mutating admission webhook

For spans to be correlated as part of the same trace, we must generate a `span context`, serialize it, and embed it in target traced objects. To accomplish this, we have introduced an [out-of-tree mutating admission webhook](https://github.com/Monkeyanator/mutating-trace-admission-controller/tree/review).

This mutating admission webhook generates a `span context`, which is the base64 encoded version of [this wire format](https://github.com/census-instrumentation/opencensus-specs/blob/master/encodings/BinaryEncoding.md#trace-context), and embeds it into the `trace.kubernetes.io/context` object annotation. The webhook can be configured to inject context into only target object types.

The decision on whether or not to sample traces from a given object is made in this webhook, upon the generation of the trace context. In addition to this, the aforementioned OpenCensus agent offers ex-post-facto trace sampling.

The proposed in-tree changes will utilize the span context annotation injected into objects with this webhook.

#### Tracing best-practices documentation

This KEP introduces a new form of instrumentation to Kubernetes, which necessiates the creation of guidelines for adding effective, standardized traces to Kubernetes components, [similar to what is found here for metrics](https://github.com/kubernetes/community/blob/master/contributors/devel/instrumentation.md).

This documentation will put forward standards for: 

* How to name spans, attributes, and annotations
* What kind of processes should be wrapped in a span
* When to link spans to other traces
* What kind of data makes for useful attributes

Having these standards in place will ensure that our tracing instrumentation works well with all backends, and that reviewers have concrete criteria to cross-check PRs against. 


### Graduation requirements

Before this proposal can be considered as a Beta feature, there are various items that must be resolved. 

1) Benchmarking the performance impact of both the in-component instrumentation and running the per-node OpenCensus agent
2) Handling owner relationships in these traces (e.g. tracing a replica set; do we append the associated pod traces to the replica set trace?)
3) Ensuring the extensibility of this tracing model to other Kubernetes objects and CRDs
4) Determining the security implications of this trace instrumentation 
5) Creation of user-facing documentation 
6) Alpha-implementation as described above

## Implementation History

* [Mutating admission webhook which injects trace context](https://github.com/Monkeyanator/mutating-trace-admission-controller)
* [Instrumentation of Kubernetes components](https://github.com/Monkeyanator/kubernetes/pull/15)<commit_msg>Expand on the decision to use the OC-agent, make various formatting changes<commit_after>---
kep-number: 0034
title: Leveraging Distributed Tracing to Understand Kubernetes Object Lifecycles
authors:
  - "@Monkeyanator"
editors:
  - "@dashpole"
owning-sig: sig-instrumentation
reviewers:
  - "@Random-Liu"
  - "@bogdandrutu"
approvers:
  - "@brancz"
creation-date: 2018-12-04
last-updated: 2018-12-07
status: provisional
---

# Leveraging Distributed Tracing to Understand Kubernetes Object Lifecycles

## Table of Contents

- [Leveraging Distributed Tracing to Understand Kubernetes Object Lifecycles](#leveraging-distributed-tracing-to-understand-kubernetes-object-lifecycles)
    - [Table of Contents](#table-of-contents)
    - [Summary](#summary)
    - [Motivation](#motivation)
        - [Goals](#goals)
        - [Non-Goals](#non-goals)
    - [Proposal](#proposal)
        - [Architecture](#architecture)
            - [Trace lifecycle](#trace-lifecycle)
            - [Root spans](#root-spans)          
            - [Context propagation](#context-propagation)          
        - [Out-of-tree changes](#out-of-tree-changes)
            - [Mutating admission webhook](#mutating-admission-webhook)
            - [Tracing best-practices documentation](#tracing-best-practices-documentation)
        - [In-tree changes](#in-tree-changes)
            - [Trace utility package](#trace-utility-package)
            - [Basic Kubernetes component trace instrumentation](#basic-kubernetes-component-trace-instrumentation)
    - [Graduation Requirements](#graduation-requirements)
    - [Implementation History](#implementation-history)

## Summary

This Kubernetes Enchancement Proposal (KEP) introduces a model for adding distributed tracing to Kubernetes object lifecycles. The inclusion of this trace instrumentation will mark a significant step in making Kubernetes processes more observable, understandable, and debuggable.


## Motivation

Debugging latency issues in Kubernetes is an involved process. There are existing tools which can be used to isolate these issues in Kubernetes, but these methods fall short for various reasons. For instance:

* **Logs**: are fragmented, and finding out which process was the bottleneck involves digging through troves of unstructured text. In addition, logs do not offer higher-level insight into overall system behavior without an extensive background on the process of interest. 
* **Events**: in Kubernetes are only kept for an hour by default, and don't integrate with visualization of analysis tools. To gain trace-like insights would require a large investment in custom tooling.
* **Latency metrics**: are gathered in some places, but these don't provide understanding into _why_ a given process was slow.

Distributed tracing, on the other hand, provides a single window into latency information from across many components and plugins. Trace data is structured, and there are numerous established backends for visualizing and querying over it.

In addition, due to the self-healing nature of Kubernetes, regressions wherein latencies are affected but the overall task is eventually accomplished are not uncommon. With our current monitoring architecture, these "soft regressions" are often difficult to observe and understand. Collecting structured trace data on per-object latencies would enable us to detect these long-term regressions automatically, and quickly determine their root causes.


### Goals

* Make it possible to visualize the progress of objects across distinct Kubernetes components
* Streamline the tedious processes involved in debugging latency issues in Kubernetes
* Make it possible to identify high-level latency regressions, and attribute them back to offending processes


### Non-Goals

* To replace existing logging, metrics, or the events API
* To trace operations from all Kubernetes resource types in a generic manner (i.e. without manual instrumentation)


## Proposal

### Architecture

#### Trace lifecycle

Kubernetes is unique in that it is constantly reconciling its actual state towards some desired state. As a result, it has no definitive concept of an "operation", which breaks the traditional model for distributed tracing. This raises the question of when to begin traces, and when to end them.

In this proposal, we choose to _only_ trace phases of an object's lifecycle wherein it's correcting from an undesired state to its desired state, and to end the trace when it enters this desired state. This means that the same object will export traces for each reconciliation it undergoes. This decision was made because:

1) These reconciliation phases are where Kubernetes performs actual work that can be traced
2) Prevents traces from spanning the entire life of an object, which can be indefninite
3) Provides a consistent, concrete duration for the root span which remains applicable across object types
4) The time required for Kubernetes controllers to reconcile desired and actual state is, by definition, a measurement of either latency experienced by end-users in the fulfillment of their requests, or down-time due to a disruption

#### Root spans

In the standard model for distributed tracing, there exists a span in each trace that all other spans are descendents of and which extends the length of the entire trace, called the `root span`. 

The Kubernetes component that kicks off an operation might not be the same component that ends it. In this proposal, when we are at the point where we want to end a root span, we craft a span to export which acts as the root span for the trace. For example, when the kubelet updates a pod from `Pending` to `Running`, it creates a root span using the start time of the pod as the start, and the current time as the end.


#### Context propagation

To correlate work done between components as belonging to the same trace, we must pass span context across process boundaries. In traditional distributed systems, this context can be passed down through RPC metadata or HTTP headers. Kubernetes, however, due to its watch-based nature, requires us to attach trace context directly to the target object. 

In this proposal, we choose to propagate this span context as an encoded string an object annotation called `trace.kubernetes.io/context`. This annotation value is regenerated and replaced when an object's trace ends, to achieve the desired behavior from [section one](#trace-lifecycle). 

This proposal chooses to use annotations as a less invasive alternative to adding a field to object metadata, but as this proposal matures, adding trace context to the official API should be considered. 


### In-tree changes

#### Vendoring the OpenCensus trace framework

This KEP proposes the use of the [OpenCensus tracing framework](https://opencensus.io/) to create and export spans to configured backends. The OpenCensus framework was chosen for various reasons: 

1) Provides concrete, tested implementations for creating and exporting spans to diverse backends, rather than providing an API specification, as is the case with [OpenTracing](https://opentracing.io/specification/)
2) [Provides an agent](https://github.com/census-instrumentation/opencensus-service) which enables lazy configuration for exporters, batching of spans, and other features 

This KEP suggests that we utilize the OpenCensus agent for the initial implementation. The alternatvies to this would be:

1) To export spans from the instrumented components themselves, not using the agent, and exposing multiple trace sinks in the in-tree tracing library, or
2) To make the agent one of the trace sinks included in the in-tree tracing library

These are all viable options, but using the OpenCensus agent makes for the least invasive in-tree changes. Before this proposal graduates through its alpha-phase, however, and is no longer on an opt-in basis, we will have to reasess whether it is viable or worthwhile to run this agent on each node across the cluster as the default.

#### Adding trace utility package

This package will be able to create spans from the span context embedded in the `trace.kubernetes.io/context` object annotation, in addition to embedding context from spans back into the annotation. This package will facilitate tracing across Kubernetes watches. It will also provide an implementation for exporting the root span for a given reconciliation trace.

This package must be divorced from specific exporters, and allow users to configure the desired tracing backend elsewhere (i.e. on the OC agent).

#### Instrumenting various Kubernetes components with pod startup traces

As we move forward with this KEP, we will use the aforementioned trace utility package to trace pod-related operations across the scheduler and kubelet. In code, this corresponds to creating a span (i.e. `ctx, span := trace.StartSpan(ctx, "Component.SampleSpan")`) at the beginning of an operation, and ending the span afterwards (`span.End()`). All calls to tracing functions will be gated with the `ObjectLifecycleTracing` alpha feature-gate, and will be disabled by default.

OpenCensus ships with plugins to transport trace context across gRPC and HTTP boundaries, which enables us to extend our tracing across the CRI and other internal boundaries.

In OpenCensus' Go implementation, span context is passed down through Go context. This will necessitate the threading of context across more of the Kubernetes codebase, which is a [desired outcome regardless](https://github.com/kubernetes/kubernetes/issues/815).

Following these intial pod-related trace additions, trace instrumentation should be added in an ad-hoc fashion to the various Kubernetes components.


### Out-of-tree changes

#### Mutating admission webhook

For spans to be correlated as part of the same trace, we must generate a `span context`, serialize it, and embed it in target traced objects. To accomplish this, we have introduced an [out-of-tree mutating admission webhook](https://github.com/Monkeyanator/mutating-trace-admission-controller/tree/review).

This mutating admission webhook generates a `span context`, which is the base64 encoded version of [this wire format](https://github.com/census-instrumentation/opencensus-specs/blob/master/encodings/BinaryEncoding.md#trace-context), and embeds it into the `trace.kubernetes.io/context` object annotation. The webhook can be configured to inject context into only target object types.

The decision on whether or not to sample traces from a given object is made in this webhook, upon the generation of the trace context. In addition to this, the aforementioned OpenCensus agent offers ex-post-facto trace sampling.

The proposed in-tree changes will utilize the span context annotation injected into objects with this webhook.

#### Tracing best-practices documentation

This KEP introduces a new form of instrumentation to Kubernetes, which necessiates the creation of guidelines for adding effective, standardized traces to Kubernetes components, [similar to what is found here for metrics](https://github.com/kubernetes/community/blob/master/contributors/devel/instrumentation.md).

This documentation will put forward standards for: 

* How to name spans, attributes, and annotations
* What kind of processes should be wrapped in a span
* When to link spans to other traces
* What kind of data makes for useful attributes

Having these standards in place will ensure that our tracing instrumentation works well with all backends, and that reviewers have concrete criteria to cross-check PRs against. 


### Graduation requirements

Before this proposal can be considered as a Beta feature, there are various items that must be resolved. 

1) Benchmarking the performance impact of both the in-component instrumentation and running the per-node OpenCensus agent
2) Handling owner relationships in these traces (e.g. tracing a replica set; do we append the associated pod traces to the replica set trace?)
3) Ensuring the extensibility of this tracing model to other Kubernetes objects and CRDs
4) Determining the security implications of this trace instrumentation 
5) Creation of user-facing documentation 
6) Alpha-implementation as described above


## Implementation History

* [Mutating admission webhook which injects trace context](https://github.com/Monkeyanator/mutating-trace-admission-controller)
* [Instrumentation of Kubernetes components](https://github.com/Monkeyanator/kubernetes/pull/15)
{'fraction_non_alphanumeric': 0.0548440065681445, 'fraction_numerical': 0.0033661740558292284, 'mean_word_length': 4.701146735314767, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 18, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '824266', 'n_tokens_mistral': 5959, 'n_tokens_neox': 5467, 'n_words': 3138}