<commit_before>---
title: Sidecar Containers
authors:
  - "@joseph-irving"
  - "@rata"
owning-sig: sig-apps
participating-sigs:
  - sig-apps
  - sig-node
reviewers:
  - "@fejta"
  - "@sjenning"
  - "@SergeyKanzhelev"
approvers:
  - "@enisoc"
  - "@kow3ns"
  - "@derekwaynecarr"
  - "@dchen1107"
editor: TBD
creation-date: 2018-05-14
last-updated: 2020-06-24
status: provisional
---

# Sidecar Containers

## Table of Contents

<!-- toc -->
- [Release Signoff Checklist](#release-signoff-checklist)
- [Summary](#summary)
- [Prerequisites](#prerequisites)
- [Motivation](#motivation)
  - [Jobs](#jobs)
  - [Startup](#startup)
  - [Shutdown](#shutdown)
- [Goals](#goals)
- [Non-Goals](#non-goals)
- [Proposal](#proposal)
  - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
    - [API Changes:](#api-changes)
    - [Kubelet Changes:](#kubelet-changes)
      - [Shutdown triggering](#shutdown-triggering)
      - [Sidecars terminated last](#sidecars-terminated-last)
      - [Sidecars started first](#sidecars-started-first)
      - [PreStop hooks sent to Sidecars first](#prestop-hooks-sent-to-sidecars-first)
    - [PoC and Demo](#poc-and-demo)
  - [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
  - [Test Plan](#test-plan)
  - [Graduation Criteria](#graduation-criteria)
    - [Alpha -&gt; Beta Graduation](#alpha---beta-graduation)
    - [Beta -&gt; GA Graduation](#beta---ga-graduation)
  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
  - [Version Skew Strategy](#version-skew-strategy)
- [Implementation History](#implementation-history)
- [Alternatives](#alternatives)
  - [Add a pod.spec.SidecarContainers array](#add-a-podspecsidecarcontainers-array)
  - [Mark one container as the Primary Container](#mark-one-container-as-the-primary-container)
  - [Boolean flag on container, Sidecar: true](#boolean-flag-on-container-sidecar-true)
  - [Mark containers whose termination kills the pod, terminationFatalToPod: true](#mark-containers-whose-termination-kills-the-pod-terminationfataltopod-true)
  - [Add &quot;Depends On&quot; semantics to containers](#add-depends-on-semantics-to-containers)
  - [Pre-defined phases for container startup/shutdown or arbitrary numbers for ordering](#pre-defined-phases-for-container-startupshutdown-or-arbitrary-numbers-for-ordering)
<!-- /toc -->

## Release Signoff Checklist

**ACTION REQUIRED:** In order to merge code into a release, there must be an issue in [kubernetes/enhancements] referencing this KEP and targeting a release milestone **before [Enhancement Freeze](https://github.com/kubernetes/sig-release/tree/master/releases)
of the targeted release**.

For enhancements that make changes to code or processes/procedures in core Kubernetes i.e., [kubernetes/kubernetes], we require the following Release Signoff checklist to be completed.

Check these off as they are completed for the Release Team to track. These checklist items _must_ be updated for the enhancement to be released.

- [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR)
- [ ] KEP approvers have set the KEP status to `implementable`
- [ ] Design details are appropriately documented
- [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
- [ ] Graduation criteria is in place
- [ ] "Implementation History" section is up-to-date for milestone
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

**Note:** Any PRs to move a KEP to `implementable` or significant changes once it is marked `implementable` should be approved by each of the KEP approvers. If any of those approvers is no longer appropriate than changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG-arch for cross cutting KEPs).

**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.

[kubernetes.io]: https://kubernetes.io/
[kubernetes/enhancements]: https://github.com/kubernetes/enhancements/issues
[kubernetes/kubernetes]: https://github.com/kubernetes/kubernetes
[kubernetes/website]: https://github.com/kubernetes/website

## Summary

To solve the problem of container lifecycle dependency we can create a new class of container: a "sidecar container" that behaves primarily like a normal container but is handled differently during termination and startup.

## Prerequisites

On June 23 2020, during SIG-node meeting, it was decided that this KEP has a
prerequisite on the (not yet submitted KEP) kubelet node graceful shutdown.

As of writing, when a node is shutdown the kubelet doesn't gracefully shutdown
all of the containers (running preStop hooks and other guarantees users would
expect). It was then decided that adding more guarantees to pod shutdown
behavior (as this KEP proposes) depends on having the kubelet gracefully
shutdown first. The main reason for this is to avoid users relying on something
we can't guarantee (like the pod shutdown sequence in case the node shutdowns).

Also, authors of this KEP and the (yet not submitted) KEP for node graceful<commit_msg>sidecar: Improve summary<commit_after>---
title: Sidecar Containers
authors:
  - "@joseph-irving"
  - "@rata"
owning-sig: sig-apps
participating-sigs:
  - sig-apps
  - sig-node
reviewers:
  - "@fejta"
  - "@sjenning"
  - "@SergeyKanzhelev"
approvers:
  - "@enisoc"
  - "@kow3ns"
  - "@derekwaynecarr"
  - "@dchen1107"
editor: TBD
creation-date: 2018-05-14
last-updated: 2020-06-24
status: provisional
---

# Sidecar Containers

## Table of Contents

<!-- toc -->
- [Release Signoff Checklist](#release-signoff-checklist)
- [Summary](#summary)
- [Prerequisites](#prerequisites)
- [Motivation](#motivation)
  - [Jobs](#jobs)
  - [Startup](#startup)
  - [Shutdown](#shutdown)
- [Goals](#goals)
- [Non-Goals](#non-goals)
- [Proposal](#proposal)
  - [Implementation Details/Notes/Constraints](#implementation-detailsnotesconstraints)
    - [API Changes:](#api-changes)
    - [Kubelet Changes:](#kubelet-changes)
      - [Shutdown triggering](#shutdown-triggering)
      - [Sidecars terminated last](#sidecars-terminated-last)
      - [Sidecars started first](#sidecars-started-first)
      - [PreStop hooks sent to Sidecars first](#prestop-hooks-sent-to-sidecars-first)
    - [PoC and Demo](#poc-and-demo)
  - [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
  - [Test Plan](#test-plan)
  - [Graduation Criteria](#graduation-criteria)
    - [Alpha -&gt; Beta Graduation](#alpha---beta-graduation)
    - [Beta -&gt; GA Graduation](#beta---ga-graduation)
  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
  - [Version Skew Strategy](#version-skew-strategy)
- [Implementation History](#implementation-history)
- [Alternatives](#alternatives)
  - [Add a pod.spec.SidecarContainers array](#add-a-podspecsidecarcontainers-array)
  - [Mark one container as the Primary Container](#mark-one-container-as-the-primary-container)
  - [Boolean flag on container, Sidecar: true](#boolean-flag-on-container-sidecar-true)
  - [Mark containers whose termination kills the pod, terminationFatalToPod: true](#mark-containers-whose-termination-kills-the-pod-terminationfataltopod-true)
  - [Add &quot;Depends On&quot; semantics to containers](#add-depends-on-semantics-to-containers)
  - [Pre-defined phases for container startup/shutdown or arbitrary numbers for ordering](#pre-defined-phases-for-container-startupshutdown-or-arbitrary-numbers-for-ordering)
<!-- /toc -->

## Release Signoff Checklist

**ACTION REQUIRED:** In order to merge code into a release, there must be an issue in [kubernetes/enhancements] referencing this KEP and targeting a release milestone **before [Enhancement Freeze](https://github.com/kubernetes/sig-release/tree/master/releases)
of the targeted release**.

For enhancements that make changes to code or processes/procedures in core Kubernetes i.e., [kubernetes/kubernetes], we require the following Release Signoff checklist to be completed.

Check these off as they are completed for the Release Team to track. These checklist items _must_ be updated for the enhancement to be released.

- [ ] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR)
- [ ] KEP approvers have set the KEP status to `implementable`
- [ ] Design details are appropriately documented
- [ ] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
- [ ] Graduation criteria is in place
- [ ] "Implementation History" section is up-to-date for milestone
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

**Note:** Any PRs to move a KEP to `implementable` or significant changes once it is marked `implementable` should be approved by each of the KEP approvers. If any of those approvers is no longer appropriate than changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG-arch for cross cutting KEPs).

**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.

[kubernetes.io]: https://kubernetes.io/
[kubernetes/enhancements]: https://github.com/kubernetes/enhancements/issues
[kubernetes/kubernetes]: https://github.com/kubernetes/kubernetes
[kubernetes/website]: https://github.com/kubernetes/website

## Summary

This KEP adds the notion of sidecar containers to Kubernetes. This KEP proses
to add a `type` field to the `containers` array in the `pod.spec` to define if a
container is a sidecar container. The only valid value for now is `sidecar`, but
other values can be added in the future if needed.

Pods with sidecar containers only change the behavior of the startup and
shutdown sequence of a pod: sidecar container are started before non-sidecars
and stopped after non-sidecars.

A pod that has sidecar containers guarantees that non-sidecar containers are
started only after sidecar containers are started and are in a ready state.
Furthermore, we propose to treat sidecar containers as regular (non-sidecar)
containers as much as possible all over the code, except for the mentioned
special startup and shutdown behavior. The rest of the pod lifecycle (regarding
restarts, etc.) remains unchanged, this KEP aims to modify only the startup and
shutdown behavior.

If a pod doesn't have a sidecar container, the behavior is completely unchanged
by this proposal.

## Prerequisites

On June 23 2020, during SIG-node meeting, it was decided that this KEP has a
prerequisite on the (not yet submitted KEP) kubelet node graceful shutdown.

As of writing, when a node is shutdown the kubelet doesn't gracefully shutdown
all of the containers (running preStop hooks and other guarantees users would
expect). It was then decided that adding more guarantees to pod shutdown
behavior (as this KEP proposes) depends on having the kubelet gracefully
shutdown first. The main reason for this is to avoid users relying on something
we can't guarantee (like the pod shutdown sequence in case the node shutdowns).

Also, authors of this KEP and the (yet not submitted) KEP for node graceful
{'fraction_non_alphanumeric': 0.1001616883669475, 'fraction_numerical': 0.004595353586928772, 'mean_word_length': 4.5988565983801815, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '808796', 'n_tokens_mistral': 3567, 'n_tokens_neox': 3236, 'n_words': 1241}